A method for guessing handwritten letters based on WiFi BFI information

Abstract

This invention discloses a method for inferring handwritten letters based on WiFi BFI information, relating to the field of WiFi side-channel attacks. The method includes: setting up a data acquisition platform to obtain BFI information when a user handwrites letters; preprocessing the acquired BFI data; extracting preprocessed features; combining data from all subcarriers to form a two-dimensional image; and saving the image as a dataset. A prediction model is trained. In real-world scenarios, the number of data samples collected from user handwriting may be relatively small, so this invention uses CGAN to expand the training set. Furthermore, adversarial learning is employed to extend the prediction model to unknown environments. Compared to traditional handwriting prediction methods based on WiFi CSI information, the proposed method does not require intrusion into the victim's device or firmware modification, making it easier to implement and deploy on a large scale.

Description

Technical Field

[0001] This invention mainly relates to the field of WiFi side-channel attacks, specifically to a method for guessing handwritten letters based on WiFi BFI information. Background Technology

[0002] WiFi is the preferred internet connectivity technology for indoor environments and is experiencing rapid growth. The global WiFi market is projected to exceed $18 billion by 2024 and expand at a CAGR of 13-15% between 2024 and 2029. Growth drivers include increasing demand for Internet of Things (IoT) devices, upgrades to WiFi technology, and the advancement of remote work and smart city projects. Today, WiFi applications extend beyond internet connectivity to include human behavior sensing, such as identity authentication, activity recognition, and health monitoring. Handwriting recognition, in particular, has wide applications in smart homes, human-computer interaction, and gaming.

[0003] Existing research has proposed various handwriting recognition methods, which can be mainly divided into four categories: camera-based methods, sensor-based methods, hardware-based methods, and WiFi-based methods. Camera-based methods use computer vision technology for handwriting recognition. However, due to limitations in line-of-sight (LOS) and lighting conditions, these methods cannot function properly in low-light environments. Motion sensor-based methods achieve high-precision recognition by wearing dedicated sensors, but users need to carry additional equipment, making them inconvenient. Hardware-based methods typically rely on dedicated devices (such as sensors and radio frequency devices) to recognize gestures, but deploying a large number of such devices in practical applications is very difficult and costly. In contrast, WiFi devices, with their widespread deployment, low cost, good wall-penetrating ability, flexible deployment methods, and ease of expansion, have shown significant potential in the field of handwriting recognition. Currently, WiFi-based handwriting recognition methods mainly utilize the channel state information (CSI) of commercial WiFi devices for analysis. However, the acquisition of CSI data faces a key challenge: the hardware capable of extracting CSI data is relatively limited. Although there are WiFi chips specifically designed for extracting CSI data, as WiFi standards are constantly updated, CSI acquisition technology also needs to be constantly adjusted. This makes WiFi sensing based on CSI technology somewhat limited in practical applications.

[0004] Therefore, we propose a method for guessing handwritten letters using WiFi BFI information. Since BFI is transmitted from smartphones to APs in plaintext, it can be eavesdropped on by any other WiFi device switched to monitoring mode, making it applicable to a wider range of scenarios. Summary of the Invention

[0005] This paper proposes a handwritten letter inference method based on WiFi BFI information. The method uses a laptop in listening mode to remotely capture BFI data frames and infers the written letter by analyzing the feature differences in BFI information when writing different letters. To address the issue of insufficient data samples, we use CGAN to augment the data. Simultaneously, we employ an adversarial learning scheme to enable the inference model to adapt to unknown scenarios.

[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0007] A method for guessing handwritten letters based on WiFi BFI information includes the following steps:

[0008] Step 1: Set up the data acquisition platform. We use a 5GHz router and a mobile phone with iperf installed to simulate sufficient traffic interaction with the router, and a laptop with monitoring mode enabled to capture BFI data.

[0009] Step 2: Extract effective features from the user's writing process;

[0010] Step 2.1: Recover the collected compressed BFI data according to the WiFi protocol, and use the recovered BFI information as the original data;

[0011] Step 2.2: Use a sliding window to remove outliers from the original data, and use the sliding window and standard deviation detection method to identify the start and end points of the handwriting activity. Collect several data points forward and backward from the start and end points of the activity to ensure that the amount of data collected is constant;

[0012] Step 2.3: Extract the information of 52 subcarriers according to the above steps, and synthesize the data of all subcarriers into a two-dimensional image to represent the changes of BFI data of each subcarrier over time. The final generated two-dimensional image is used as the original input data of the classification model.

[0013] Step 3: To address the limited number of BFI information data samples for handwritten letters collected, a Conditional Generative Adversarial Network (CGAN) is used to augment the training set.

[0014] Step 3.1: Divide the original dataset into a training set and a test set, with a ratio of 7:3 between the training set and the test set, and use the training set to train CGAN;

[0015] Step 3.2: Use the trained generator to generate 50 new data samples for each handwritten letter, and add the newly generated samples to the training set to expand the training set;

[0016] Step 4: Solve the problem of classification models' dependence on the environment based on adversarial learning;

[0017] Step 4.1: Collect BFI data samples of handwritten letters from different environments, and retain data from one environment to test the robustness of the classification model to different environments;

[0018] Step 4.2: For data samples collected in other environments, use the method in Step 3 to divide the dataset and expand the training set;

[0019] Step 4.3: Use a Convolutional Neural Network (CNN) to extract features from the input data, and introduce an environment classification model and an alphabet classification model to classify the extracted features. Set the loss function to the classification accuracy of the alphabet classification model minus k times the classification accuracy of the environment classification model. Through continuous learning, the extracted features are made to contain as little environmental information as possible.

[0020] Step 5: Performance testing of the handwritten letter inference model;

[0021] Step 5.1: After completing step 4, retain the trained feature extraction module and letter classification model. The environment classification model is no longer needed during the inference process.

[0022] Step 5.2: Use the previously defined test set to test the classification accuracy of the letter classification model;

[0023] Step 5.3: Test the environmental robustness of the classification model using data samples from a specific untrained environment retained in Step 4.1;

[0024] The features of our proposed method are:

[0025] (1) Our inference method is non-intrusive and only requires a laptop with listening mode enabled to capture BFI data frames, thereby inferring the user's handwritten letters, which has wide applicability.

[0026] (2) By using CGAN to augment the training set, the problem of insufficient samples of handwritten 26 letters in real life was solved.

[0027] (3) Through adversarial learning, the problem of the classification model's dependence on the environment is solved, enabling the classification model to be extended to unknown scenarios.

[0028] Therefore, this method, as a novel handwritten letter guessing method, uses WiFi BFI information as the sole input for handwritten letter guessing, which can effectively supplement and expand existing handwritten letter guessing technologies and is suitable for a wider range of application scenarios. Attached Figure Description

[0029] Figure 1 Diagram of the proposed handwritten letter inference system

[0030] Figure 2 Flowchart of the proposed inference method

[0031] Figure 3 The original time-domain plot of BFI and the time-domain plot after outlier processing.

[0032] Figure 4 The start and end regions of handwriting activity detected based on sliding window and standard deviation.

[0033] Figure 5 Network structure diagram for adversarial learning

[0034] Figure 6 The graph shows the average classification accuracy of each letter in ten experiments.

[0035] Experimental setup

[0036] This invention selects places such as laboratories, offices, and cafes to collect data and conduct experiments. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings.

[0038] Reference Figure 1 and Figure 2 A method for guessing user handwritten letters based on WiFi BFI information; the data collection platform consists of a router, a mobile phone, and a laptop.

[0039] The inference process consists of a data acquisition phase, a data processing phase, a data augmentation phase, an adversarial training phase, and a verification phase.

[0040] Data collection phase:

[0041] Step 1. Set the router to 5G band, channel 52, and enable Beamforming directional signal transmission and MU-MIMO;

[0042] Step 2. Connect your phone to the router network, enable iperf to simulate sufficient traffic interaction, and use a laptop with monitoring mode enabled to capture WiFi data frames using Wireshark software.

[0043] Step 3. Capture NO ACK data frames during the period from when the user is at rest to when they start writing, and save the captured data in JSON format;

[0044] Step 4. In order to test the performance of the classification model in terms of environmental robustness, we collected data in three different environments, collecting 40 data samples of each letter in each environment.

[0045] Data processing stage:

[0046] Step 5. Extract the BFI information stored in the JSON file, and restore the compressed BFI data to its original form according to the WiFi protocol as the initial data;

[0047] Step 6. (As shown) Figure 3 As shown, observing the original data reveals that most data points remain flat when the user is stationary, but occasional outliers exist. Therefore, we use a sliding window to remove outliers from the original data. The waveform after removing outliers is also shown in the image. Figure 3 It was shown in the middle;

[0048] Step 7. Figure 4 As shown, in order to obtain the start and end points of the user's writing activity, we use a sliding window and standard deviation for detection. When the standard deviation of the data in the window exceeds the set threshold, the start point of the window is set as the start point of an activity. After determining the start point, when the standard deviation of the data in the window is less than the set threshold, the end point of the window is set as the end point of an activity.

[0049] Step 8. Since the temporal data length of the handwriting activity is not fixed, but we want the feature data size of each letter to remain consistent, after removing outliers using a sliding window and detecting the start and end times of the handwriting activity based on the standard deviation, we extract several data points forward and backward from the start and end of the activity, respectively, to ensure that the feature data length extracted for each letter is consistent. If the original data length is insufficient, missing data is padded with 0;

[0050] Step 9. Extract the time-domain data of 52 subcarriers following the steps above. Then, combine the information of all subcarriers into a two-dimensional image. Specifically, the horizontal axis of the two-dimensional image represents time, the vertical axis represents the subcarrier index, and the color intensity of the image represents the data size. The resulting two-dimensional image will ultimately be used as the dataset for model training and testing.

[0051] Data expansion phase:

[0052] Step 10. In order to improve the performance of the classification model, we use CGAN to augment the training set, given the limited number of handwritten letter data samples collected.

[0053] Step 11. First, divide the collected dataset into a training set and a test set in a ratio of 7:3. Set parameters such as the number of iterations and the learning rate of the model, and train CGAN using the training set.

[0054] Step 12. After training is complete, keep the CGAN generator, use the generator to generate 50 new data samples for each handwritten letter, and add them to the original training set;

[0055] Model training phase:

[0056] Step 13. Figure 5 As shown, our network structure includes a feature extraction module, an environment classification model, and an letter classification model. To make the letter classification model applicable to different environments, we define the loss function of the entire model as the classification accuracy of the letter classification model minus k times the classification accuracy of the environment classification model. During continuous adversarial training, this allows the feature extraction module to extract as few environment-related features as possible.

[0057] Step 14. After setting the parameters such as the number of iterations and the learning rate, train the model using the training set, and retain the trained feature extraction module and letter classification module as the final handwritten letter inference model;

[0058] Verification phase:

[0059] We used the test set obtained in step 3 to test the accuracy of the letter classification model. The average classification accuracy of each letter across ten experiments is as follows: Figure 6 As shown;

[0060] To verify the model's robustness to different environments, we collected data samples of users writing letters in multiple environments, tested the model's performance using leave-one-out validation, and compared the performance of the model without adversarial learning.

[0061] The handwriting inference method based on WiFi BFI information proposed in this invention differs significantly from traditional handwriting recognition methods. Firstly, most traditional handwriting recognition methods rely on cameras, sensors, hardware devices, and WiFi CSI for identification. Camera-based methods utilize computer vision technology for handwriting recognition. However, due to limitations in line of sight (LOS) and brightness, these methods cannot operate in dark environments. In motion sensor-based implementations, high-precision recognition can be achieved by wearing dedicated sensors, but this is inconvenient for users. Existing hardware-based systems primarily use dedicated hardware to recognize gestures, but achieving this through the deployment of numerous sensor nodes and dedicated radio frequency devices is practically impractical. Utilizing CSI data can achieve good recognition results to a certain extent. However, CSI data acquisition faces a key challenge: hardware that supports firmware modification for CSI data extraction remains limited. Even with the development of dedicated WiFi chips for CSI extraction, CSI acquisition technology needs to be adjusted accordingly with updates to WiFi standards. This limits the practical application of WiFi sensing based on CSI technology. Therefore, this invention not only overcomes the limitations of traditional technologies in complex environments, but also improves the practicality of handwriting inference, especially by inferring the user's writing content without relying on dedicated hardware or modifying firmware, making handwriting inference more widely applicable in real-world scenarios.

Claims

1. A method for predicting handwritten letters based on WiFi BFI information, characterized in that, Comprising the following steps: Step 1: build a data collection platform, build a platform for collecting data and the process of collecting data is as follows: Step 1.1: the data collection platform includes a router, a mobile phone and a notebook computer, the router is set to 5G frequency 52 channels, and the Beamforming directional signal transmission and MU-MIMO are turned on; Step 1.2: connect the mobile phone to the router network, start the iperf simulation of sufficient traffic interaction, and use a notebook computer with listening mode enabled to capture WiFi data frames using Wireshark software; Step 1.3: capture the NO ACK data frames during the user's transition from static to writing activity, which contain BFI information, and save the captured data as json format; Step 1.4: in order to test the performance of the classification model in subsequent tests, we collect 40 data samples for each letter in multiple environments; Step 2: extract effective features during user writing; Step 3: in view of the small amount of BFI data samples collected for handwritten letters, CGAN is used to expand the training set; Step 4: based on the dependence of the classification model on the environment, the dependence of the handwritten letter prediction model on the environment is solved based on the specific steps of the adversarial learning: Step 4.1: the adversarial learning network structure includes a feature extraction module, an environment classification model and a letter classification model; in order to make the letter classification model suitable for different environments, the loss function of the whole model is defined as the classification accuracy of the letter classification model minus k times the classification accuracy of the environment classification model, and in the continuous adversarial training process, the feature extraction module can extract as few environment related features as possible; Step 4.2: after setting the iteration number and learning rate parameters, the model is trained using the training set, and the trained feature extraction module and letter classification module are reserved as the final handwritten letter prediction model; Step 5: performance verification of the handwritten letter prediction model. 2.The handwriting letter inference method based on WiFi BFI information according to claim 1, wherein, The specific steps for extracting effective features during user writing are as follows: Step 2.1: extract the BFI information in the saved json file, and according to the WiFi protocol, restore the compressed BFI data to the original form as the initial data; Step 2.2: use a sliding window to remove outliers in the original data, and based on the sliding window and standard deviation, detect the start and end points of handwriting activity, then extract a number of data points from the start point and a number of data points from the end point respectively, to ensure that the number of data features extracted for each handwritten letter is consistent; Step 2.3: assemble the time domain data of 52 subcarriers into a two-dimensional image after preprocessing, and save the assembled data set as a two-dimensional image. 3.The method of handwriting letter inference based on WiFi BFI information according to claim 1, characterized in that, The specific steps for expanding the training set using CGAN are as follows: Step 3.1: first, divide the collected data set into training set and test set, the ratio is 7:3, set the iteration number and learning rate parameters of the model, and use the training set to train CGAN; Step 3.2: After the training is completed, the generator of the CGAN is retained, and the generator is used to generate 50 new data samples for each handwritten letter and added to the original training set. 4.The method of handwriting letter inference based on WiFi BFI information according to claim 1, characterized in that, The specific steps for verifying the performance of the handwritten letter inference model are as follows: Step 5.1: Test the model's classification accuracy using the test set, and calculate the classification accuracy of each letter in multiple experiments; Step 5.2: To verify the robustness of the handwritten letter inference model to the environment, we collect handwritten letter BFI data samples under multiple environments, test the model's performance using the leave-one-out validation method, and compare the performance of the model without using adversarial learning.

Images