Handwritten letter speculation method based on WiFi BFI information

Abstract

The invention discloses a handwriting letter speculation method based on WiFi BFI information, and relates to the field of WiFi side channel attacks. The method comprises the steps of setting a data acquisition platform, obtaining BFI information when a user writes letters, preprocessing the acquired BFI data, extracting features after preprocessing, combining data of all subcarriers to form a two-dimensional image, and storing the two-dimensional image as a data set. According to the method, the prediction model is trained, in an actual scene, the number of data samples collected when a user writes is small, so that a training set is expanded by using the CGAN, and in addition, the prediction model can be popularized to be used in an unknown environment by adopting adversarial learning. Compared with a traditional method for carrying out handwriting speculation based on WiFi CSI information, the speculation method provided by the invention does not need to invade damaged equipment, does not need to modify firmware, and is easier to implement and deploy in a large scale.

Description

Technical Field

[0001] The present invention mainly relates to the field of WiFi side channel attacks, and in particular to a handwritten letter inference method based on WiFi BFI information. Background Art

[0002] WiFi is the preferred Internet connection technology in indoor environments and is growing rapidly. The global WiFi market is expected to exceed US$18 billion by 2024 and expand at a compound annual growth rate of 13-15% between 2024 and 2029. Growth drivers include increased demand for Internet of Things (IoT) devices, upgrades to WiFi technology, and the advancement of remote work and smart city projects. Today, the application of WiFi is not limited to Internet connection, but is also widely used in human behavior perception, such as identity authentication, activity recognition, and health monitoring. Among them, handwriting recognition has a wide range of applications in smart homes, human-computer interaction, and games.

[0003] Existing studies have proposed a variety of handwriting recognition methods, which can be mainly divided into four categories: camera-based methods, sensor-based methods, hardware-based methods, and WiFi-based methods. Camera-based methods use computer vision technology to perform handwriting recognition. However, due to the limitations of line of sight (LOS) and lighting conditions, this method does not work properly in dim environments. Motion sensor-based methods achieve high-precision recognition by wearing dedicated sensors, but users need to carry additional equipment, which is inconvenient to use. Hardware-based methods usually rely on dedicated equipment (such as sensors and radio frequency devices) to recognize gestures, but in practical applications, it is very difficult and costly to deploy a large number of such devices. In contrast, WiFi devices have shown significant potential in the field of handwriting recognition due to their wide deployment, low cost, good wall penetration ability, flexible deployment methods, and easy scalability. At present, WiFi-based handwriting recognition methods mainly use the channel state information (CSI) of commercial WiFi devices for analysis. However, the collection of CSI data faces a key challenge: the hardware that can extract CSI data is relatively limited. Although there are WiFi chips specifically designed to extract CSI data, as WiFi standards are constantly updated, CSI collection technology also needs to be constantly adjusted, which makes WiFi perception based on CSI technology have certain limitations in practical applications.

[0004] Therefore, we propose a method to infer handwritten letters using WiFi BFI information. Since BFI is transmitted from a smartphone to the AP in plain text, it can be eavesdropped by any other WiFi device switched to monitoring mode, and is applicable to a wider range of scenarios. Summary of the invention

[0005] This paper proposes a handwritten letter inference method based on WiFi BFI information. This method uses a laptop in monitoring mode to remotely capture BFI data frames and infer the written letters by the feature differences of BFI information when writing different letters. To address the problem of insufficient data samples, we use CGAN for data augmentation. At the same time, we use an adversarial learning scheme to enable the inference model to adapt to unknown scenarios.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned object is:

[0007] A handwritten letter estimation method based on WiFi BFI information comprises the following steps:

[0008] Step 1: Set up the data collection platform. We use a 5GHz router and a mobile phone with the iperf tool installed to simulate sufficient traffic interaction with the router, and use a laptop with monitoring mode turned on to capture BFI data.

[0009] Step 2: Extract effective features during the user's writing process;

[0010] Step 2.1: Restore the compressed BFI data collected according to the WiFi protocol, and use the restored BFI information as the original data;

[0011] Step 2.2: Use a sliding window to remove outliers from the original data, and use the sliding window and standard deviation detection method to identify the start and end points of the handwriting activity. Collect several data points forward and backward from the start and end points of the activity to ensure that the amount of data collected is fixed;

[0012] Step 2.3: Extract the information of 52 subcarriers according to the above steps, and synthesize the data of all subcarriers into a two-dimensional image to show the change of BFI data of each subcarrier when time changes. The final generated two-dimensional image is used as the original input data of the classification model;

[0013] Step 3: In view of the fact that the collected BFI information data samples of handwritten letters are small, the conditional generative adversarial network (CGAN) is used to expand the data of the training set;

[0014] Step 3.1: Divide the original dataset into a training set and a test set, with a ratio of 7:3 between the training set and the test set, and use the training set to train CGAN;

[0015] Step 3.2: Use the trained generator to generate 50 new data samples for each handwritten letter, and add the newly generated samples to the training set to expand the training set;

[0016] Step 4: Solve the problem of classification model's dependence on the environment based on adversarial learning;

[0017] Step 4.1: Collect BFI data samples of handwritten letters from different environments, and keep the data in one environment to test the robustness of the classification model to different environments;

[0018] Step 4.2: For the data samples collected in the remaining environments, use the method in step 3 to divide the data set and expand the training set;

[0019] Step 4.3: Use a convolutional neural network (CNN) to extract features of the input data, and introduce an environmental classification model and a letter classification model to classify the extracted features. Set the loss function to the classification accuracy of the letter classification model minus k times the classification accuracy of the environmental classification model. Through continuous learning, the extracted features are made to contain as little environmental information as possible.

[0020] Step 5: Performance test of the handwritten letter inference model;

[0021] Step 5.1: After step 4 is completed, the trained feature extraction module and letter classification model are retained, and the environment classification model is no longer needed in the inference process;

[0022] Step 5.2: Use the previously divided test set to test the classification accuracy of the letter classification model;

[0023] Step 5.3: Use the data sample from a certain untrained environment reserved in step 4.1 to test the environmental robustness of the classification model;

[0024] The features of our proposed method are:

[0025] (1) Our inference method is non-invasive and only requires a laptop in monitoring mode to capture BFI data frames and infer the user's handwritten letters, which has wide applicability.

[0026] (2) By using CGAN to expand the training set data, the problem of insufficient samples of 26 handwritten letters in real life is solved.

[0027] (3) Through adversarial learning, the problem of the classification model’s dependence on the environment is solved, allowing the classification model to be extended to unknown scenarios.

[0028] Therefore, as a new handwritten letter inference method, this method uses WiFi BFI information as the only input for handwriting inference, which can effectively supplement and expand the existing handwritten letter inference technology and is suitable for a wider range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Diagram of the proposed handwritten letter inference system

[0030] Figure 2 Flow chart of the proposed inference method

[0031] Figure 3 The original time domain diagram of BFI and the time domain diagram after processing abnormal points

[0032] Figure 4 The start and end areas of handwriting activity detected based on sliding window and standard deviation

[0033] Figure 5 Network structure diagram for adversarial learning

[0034] Figure 6 The average classification accuracy of each letter in ten experiments is shown in the figure Experimental setup

[0035] The present invention selects to collect data and conduct experiments in laboratories, offices, cafes and the like. DETAILED DESCRIPTION

[0036] The present invention is further described below in conjunction with the accompanying drawings.

[0037] Reference Figure 1 and Figure 2 ,A method for estimating user handwritten letters based on WiFi BFI information, the data collection platform consists of routers, mobile phones, and laptops;

[0038] The inference process consists of the data collection phase, data processing phase, data expansion phase, adversarial training phase, and verification phase;

[0039] Data collection phase:

[0040] Step 1. Set the router to 5G band channel 52 and enable Beamforming directional signal transmission and MU-MIMO;

[0041] Step 2. Connect your phone to the router network, open iperf to simulate sufficient traffic interaction, and use a laptop computer in monitoring mode to capture WiFi data frames using Wireshark software;

[0042] Step 3. Capture the NO ACK data frame when the user is still and writing, and save the captured data in json format;

[0043] Step 4. In order to subsequently test the performance of the classification model in terms of environmental robustness, we collected data in three different environments, collecting 40 data samples of each letter in each environment;

[0044] Data processing stage:

[0045] Step 5. Extract the BFI information saved in the json file, and restore the compressed BFI data to its original form as the initial data according to the WiFi protocol;

[0046] Step 6. Figure 3 As shown in Figure 2, by observing the original data, we can find that when the user is stationary, most of the data points are flat, but there are occasional abnormal points. Therefore, we use a sliding window to remove the abnormal points in the original data. The waveform after removing the abnormal points is also Figure 3 It is shown in;

[0047] Step 7. Figure 4 As shown in the figure, in order to obtain the starting and ending points of the user's writing activities, we use sliding windows and standard deviations for detection. When the standard deviation of the data in the window exceeds the set threshold, the starting point of the window is set as the starting point of an activity. After determining the starting point, when the standard deviation of the data in the window is less than the set threshold, the end point of the window is set as the end point of an activity.

[0048] Step 8. Since the length of the time series data of handwriting activities is not fixed, but we want the feature data size of each letter to be consistent, we use a sliding window to remove outliers and detect the start and end time of the handwriting activity based on the standard deviation. Then, we extract several data points forward and backward from the start and end of the activity to ensure that the length of the feature data extracted for each letter is consistent. If the original data length is insufficient, the missing data is supplemented with 0;

[0049] Step 9. Extract the time domain data of 52 subcarriers according to the above steps, and then combine the information of all subcarriers into a two-dimensional image. Specifically, the horizontal axis of the two-dimensional image represents time, the vertical axis represents the index of the subcarrier, and the color intensity of the image represents the data size. Finally, the obtained two-dimensional image is used as the data set for model training and testing;

[0050] Data expansion phase:

[0051] Step 10. In order to improve the effect of the classification model, we use CGAN to expand the training set data when the collected handwritten letter data samples are small;

[0052] Step 11. First, divide the collected data set into training set and test set with a ratio of 7:3, set the parameters such as the number of model iterations and learning rate, and use the training set to train CGAN;

[0053] Step 12. After training, keep the CGAN generator and use it to generate 50 new data samples for each handwritten letter and add them to the original training set;

[0054] Model training phase:

[0055] Step 13. Figure 5 As shown in the figure, our network structure includes a feature extraction module, an environment classification model and a letter classification model. In order to make the letter classification model applicable to different environments, we define the loss function of the entire model as the classification accuracy of the letter classification model minus k times the classification accuracy of the environment classification model. In the process of continuous adversarial training, the feature extraction module can extract as few environment-related features as possible;

[0056] Step 14. After setting parameters such as the number of iterations and learning rate, use the training set to train the model, and retain the trained feature extraction module and letter classification module as the final handwritten letter inference model;

[0057] Verification phase:

[0058] We use the test set divided in step 3 to test the accuracy of the letter classification model. The average classification accuracy of each letter in ten experiments is as follows: Figure 6 As shown;

[0059] In order to verify the robustness of the model to the environment, we collected data samples of user-written letters in multiple environments, tested the performance of the model using the leave-one-out validation method, and compared the performance of the model without adversarial learning.

[0060] The handwriting inference method based on WiFi BFI information proposed in the present invention is significantly different from the traditional handwriting recognition method. First, most of the traditional handwriting recognition methods are based on cameras, sensors, hardware devices and WiFi CSI for recognition, and the camera-based method uses computer vision technology for handwriting recognition. However, due to the limitations of line of sight (LOS) and brightness, this method cannot work in dark places. In the implementation method based on motion sensors, high-precision recognition can be obtained by wearing dedicated sensors, but it is very inconvenient for users to carry. Existing systems based on special hardware mainly use dedicated hardware to recognize gestures, and it is not feasible to achieve this goal by deploying a large number of sensor nodes and dedicated radio frequency devices in practice. Using CSI data can achieve good recognition effects to a certain extent. However, the collection of CSI data faces a key challenge: the hardware that supports modifying firmware to extract CSI data is still limited. Even if a WiFi chip specifically for extracting CSI is developed, as the WiFi standard is updated and iterated, the CSI acquisition technology also needs to be adjusted accordingly. This makes WiFi perception based on CSI technology have certain limitations in practical applications. Therefore, the present invention not only overcomes the limitations of traditional technologies in complex environments, but also improves the practicality of handwriting speculation, especially inferring the user's writing content without relying on dedicated hardware or modified firmware, so that handwriting speculation can be more widely used in actual scenarios.

Claims

1. A handwritten letter estimation method based on WiFi BFI information, comprising the following steps: Step 1: Build a data collection platform; Step 2: Extract effective features during the user's writing process; Step 3: In view of the small number of collected handwritten BFI data samples, a conditional generative adversarial network (CGAN) is used to expand the training set data; Step 4: Solve the problem of the classification model's dependence on the environment based on adversarial learning; Step 5: Verify the performance of the handwritten letter inference model.

2. According to claim 1, the platform for collecting data and the process of collecting data are specifically as follows: Step 1.1: The data collection platform includes a router, a mobile phone, and a laptop. The router is set to 5G band channel 52, and Beamforming directional signal transmission and MU-MIMO are enabled. Step 1.2: Connect your phone to the router network, open iperf to simulate sufficient traffic interaction, and use a laptop in monitoring mode to capture WiFi data frames using Wireshark software; Step 1.3: Capture the NO ACK data frame during the user's transition from static to writing activity, which contains BFI information, and save the captured data in JSON format; Step 1.4: In order to subsequently test the performance of the classification model in terms of environmental robustness, we collected data in multiple environments, collecting 40 data samples for each letter.

3. According to claim 1, the specific steps of extracting effective features in the user's writing process are: Step 2.1: Extract the BFI information in the saved json file, and restore the compressed BFI data to its original form as the initial data according to the WiFi protocol; Step 2.2: Use a sliding window to remove outliers in the original data, and detect the start and end points of the handwriting activity based on the sliding window and standard deviation. Then, extract several data points from the start point forward and from the end point backward to ensure that the number of data features extracted for each handwritten letter is consistent; Step 2.3: The time domain data of 52 subcarriers are preprocessed and assembled into a two-dimensional image, and all collected data are assembled into a two-dimensional image and saved as a data set.

4. According to claim 1, the specific steps of using CGAN to expand the training set are: Step 3.1: First, divide the collected data set into training set and test set with a ratio of 7:3, set the model parameters such as the number of iterations and learning rate, and use the training set to train CGAN; Step 3.2: After training, keep the generator of CGAN and use it to generate 50 new data samples for each handwritten letter and add them to the original training set.

5. According to claim 1, the specific steps of solving the problem of the dependence of the handwritten letter inference model on the environment based on adversarial learning are: Step 4.1: The adversarial learning network structure includes a feature extraction module, an environment classification model and a letter classification model. In order to make the letter classification model applicable to different environments, we define the loss function of the entire model as the classification accuracy of the letter classification model minus k times the classification accuracy of the environment classification model. In the continuous adversarial training process, the feature extraction module can extract as few environment-related features as possible; Step 4.2: After setting parameters such as the number of iterations and learning rate, use the training set to train the model, and retain the trained feature extraction module and letter classification module as the final handwritten letter inference model.

6. According to claim 1, the specific steps of verifying the performance of the handwritten letter inference model are: Step 5.1: Use the test set to test the classification accuracy of the model, and count the classification accuracy of each letter in multiple experiments; Step 5.2: In order to verify the robustness of the handwritten letter inference model to the environment, we collected handwritten letter BFI data samples from multiple environments, used the leave-one-out validation method to test the performance of the model, and compared the performance of the model without adversarial learning.

Images