A keyboard keystroke recognition system and method based on WiFi network signals

Abstract

The application discloses a keyboard keystroke recognition system and method based on a WiFi network signal and belongs to the field of human-computer interaction. The system comprises a CSI data acquisition hardware platform and a keyboard. The CSI data acquisition hardware platform comprises a WiFi signal transmitting end and a WiFi signal receiving end. The WiFi signal transmitting end and the receiving end are respectively connected with three antennas. The three antennas connected with the transmitting end are all located on the right side of the keyboard, and the three antennas connected with the receiving end are all located on the left side of the keyboard. A trained WiNum neural network model is further installed in the WiFi signal receiving end. The method can infer whether keystrokes occur and the categories of the keystrokes by processing the CSI data collected by the data acquisition hardware platform by using the designed keystroke recognition algorithm. The application has high recognition accuracy and stability and does not need the assistance of other sensors, greatly reduces the development cost, has high portability, and in addition, the system is simple to install and maintain and can operate for a long time.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of human-computer interaction technology, and particularly relates to a keyboard keystroke recognition system and method based on WiFi network signals. BACKGROUND

[0002] With the progress of the times and the development of science and technology, people's requirements for the quality of life are getting higher and higher, and the degree of intelligence of the devices used is also getting higher and higher. Portable devices such as notebooks, smart phones, tablet computers and the like are increasingly centralized in function and miniaturized. However, the traditional keyboard has the defects of large size and inconvenience for carrying, and the mobile phone keyboard is limited by the small size of the keys and is not convenient for operation. These hinder the comfort and portability of human-computer interaction. Emerging human-computer interaction technology (HCI) is an effective means to solve this problem. Keystroke recognition technology, as an important part of HCI, has attracted more and more attention of researchers.

[0003] Currently, the common keystroke recognition techniques are mainly divided into four categories: one is the keystroke recognition based on wearable devices, which collects the motion data of gestures by wearing devices containing specific sensors to determine the state of gestures. Two is the keystroke recognition based on image processing technology, which collects video or image of gesture information and detects keystroke motion by image processing. Three is the keystroke recognition based on sound positioning, which obtains the sound signal of the user's desktop tapping through the microphone and accurately calculates the time difference to accurately identify the tapping position of the user. Four is the keystroke recognition based on WiFi, which obtains the information of gesture changes through specific WiFi receiving and sending devices. The keystroke recognition based on wearable devices requires the target person to actively wear the specified auxiliary equipment, which is not conducive to use. The keystroke recognition based on vision relies on high-resolution video or image under visual distance, which is easily disturbed by light and hand posture, and in some scenarios it easily infringes on the privacy of the user and is high in cost. The keystroke recognition based on sound positioning has the problem of low detection accuracy because the input signal is easily disturbed. Because the radio signal has the advantages of strong penetration, large sensing range, and no recording of sensitive information about privacy. More importantly, with the development of technology, cellular mobile signals, television tower signals, and WiFi infrastructure are ubiquitous. Touchless perception based on radio signals is the most ideal form of perception in current pervasive computing. Keystroke detection based on radio signals is of great significance to the development of HCI technology. Compared with traditional physical keyboards, it is more convenient, more space-saving, lower in cost, and more technologically advanced. For example, in public places, keystroke recognition technology replaces physical keyboards to achieve non-contact input, thereby reducing the spread of viruses. In the process of driving a car, remotely operating a mobile phone makes driving safer. In smart home, the input of various intelligent devices such as televisions and air conditioners is completed by keystroke recognition technology, which is more convenient and space-saving. Most of the current recognition techniques based on wireless signals recognize relatively large actions, and their features are relatively obvious and easy to extract, but the recognition of keystrokes, which are small actions, is still a difficulty. In addition to the great application value of keystroke recognition technology based on radio signals, it can also be extended to sign language recognition, medical care, and old-age services based on small action recognition of radio signals, so it has far-reaching significance. Keystroke recognition technology based on radio signals has great foresight and research significance. In 2015, Ali Kt proposed the WiKey recognition system, which first proved the possibility of using commercial WiFi devices to recognize micro-gestures such as keystrokes. The system uses commercial WiFi devices to detect the action of tapping the keyboard with fingers, and detects the tapped key by analyzing the change of CSI amplitude caused by tapping the keyboard. In 2016, Professor Zhu Haojin's team of Shanghai Jiaotong University developed the WindTalker system.It uses public WiFi to collect user CSI data and implements windtalker on several mobile phones, with an identification accuracy of about 75%. In 2021, A.A.Arafat et al. used WiFi signals to identify virtual keys in VR headsets for the first time. They designed the VR-Spy system, which implements VR-Spy using two commercial off-the-shelf (COTS) devices, a transmitter (WAVLINK router) and a receiver (Intel NUC with IWL 5300 network card). VR-Spy achieves a virtual keystroke recognition accuracy of 69.75%. However, the above method has the problems of low recognition accuracy and weak anti-interference ability, and the present application provides a solution to these problems. SUMMARY

[0004] The purpose of the present application is to provide a keyboard keystroke recognition system and method based on WiFi network signals, which is reasonable in design, overcomes the shortcomings of the prior art, and has good effects.

[0005] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0006] A keyboard keystroke recognition system based on WiFi network signals, comprising a CSI data acquisition hardware platform and a keyboard; the CSI data acquisition hardware platform comprises a WiFi signal transmitting end and a WiFi signal receiving end; the left and right sides of the keyboard are respectively provided with an antenna connected to the WiFi signal receiving end and an antenna connected to the WiFi signal transmitting end;

[0007] The WiFi signal transmitting end is configured to transmit WiFi signals.

[0008] The WiFi signal receiving end is configured to receive data and record CSI data with keystroke action information.

[0009] Further, the WiFi signal transmitting end uses a notebook computer or a commercial wireless router, the transmitting end is connected to three antennas arranged in a straight line, the antennas are evenly distributed, the interval is 40 cm, the distance from the keyboard right edge is 120 cm, and the antenna placement position is 10 cm higher than the keyboard.

[0010] Further, the WiFi signal receiving end uses a desktop computer equipped with a network card, the receiving end is connected to three antennas, and the three antennas are numbered 0, 1 and 2 from bottom to top around the keyboard, wherein the 0th antenna is 30 cm away from the keyboard, the 1st antenna is 20 cm away from the keyboard, and the 2nd antenna is 7 cm away from the keyboard.

[0011] Further, the antenna uses a 5dB omnidirectional antenna, and the three antennas connected to the receiving end are bent to 120 degrees.

[0012] Further, the WiFi signal receiving end is installed with a trained WiNum neural network model, and the WiNum neural network model adopts a WiNum algorithm, and the data collected by the CSI data collection hardware platform is processed by the WiNum algorithm to obtain the time of the keystroke action and the category of the keystroke.

[0013] A keyboard keystroke recognition method based on a WiFi network signal, which adopts the keyboard keystroke recognition system based on the WiFi network signal as described above, and identifies the position and category of the keystroke by the WiNum algorithm; the method specifically includes the following contents:

[0014] Stage one, model training stage; the specific process is as follows:

[0015] Step 1.1, build a CSI data collection hardware platform, after building, the user repeatedly strikes the keys on the keyboard to be identified, and collects the training data required by the WiNum neural network model;

[0016] Step 1.2, use the collected data to perform offline training test on the WiNum neural network model;

[0017] Before training, the collected data is preprocessed, the required CSI amplitude information is separated from the collected data, and the data is labeled and divided into training and test sets;

[0018] Then, the training set is put into the neural network model written according to the WiNum algorithm for multiple iterations to gradually optimize the model parameters;

[0019] Step 1.3, after the model training is completed, test with the test set to check the recognition effect, if the model recognition effect is good, output the model, if the recognition effect is not good, adjust the model parameters and retrain until the model recognition effect is good;

[0020] Stage two, online recognition stage;

[0021] Step 2.1, install the trained WiNum neural network model into the WiFi signal receiving end; when the user strikes the keyboard, the corresponding CSI data is collected, and the data is preprocessed to eliminate the noise contained in part of the information; the specific process is as follows:

[0022] Step 2.1.1, Butterworth low-pass filter is used to remove high-frequency interference, and the cut-off frequency w c is obtained by the following formula:

[0023]

[0024] Wherein, f represents the highest frequency of the useful signal, F represents the signal sampling frequency;

[0025] Step 2.1.2, normalization processing is carried out;

[0026] Step 2.1.3, PCA filtering is carried out, and the first three principal components of the data signal are taken;

[0027] Step 2.2, keystroke action detection; the specific process is as follows:

[0028] Step 2.2.1, three receiving antennas and three transmitting antennas are combined in pairs to form nine antenna pairs, and the sliding window processing is carried out on each principal component sequence of each antenna pair, and the calculation formula is as follows:

[0029]

[0030] Wherein, w represents the length of the sliding window, s i k represents the kth principal component sequence of the i antenna pair, j represents the position of the sequence point, m i k represents the sliding window processing result of the kth principal component sequence of the i antenna pair;

[0031] Step 2.2.2, the principal component sequence after sliding window processing of each antenna pair is added to form sequence S1;

[0032] Step 2.2.3, first-order difference processing is carried out on S1 to obtain D1, and the calculation formula is as follows:

[0033] d i (j)=s i (j+1)-s i (j) (3) Wherein, s i (j) represents the jth sample point of the i antenna pair, d i (j) represents the difference processing result of the j position of the i antenna pair;

[0034] Step 2.2.4, a threshold K is set, if D1 exceeds K for a certain number of times, it is considered that keystroke occurs, and the point is recorded as p0 point, otherwise, the current point is not recorded;

[0035] Step 2.3, keystroke position detection; the specific process is as follows:

[0036] Step 2.3.1, similar to obtaining S1, the length of the sliding window is changed to half of that when processing S1 to obtain S2;

[0037] Step 2.3.2, from the position of p0 point, when the position of sequence S2 continuously increases or decreases for a certain number of times, the position is recorded as p1, then it is considered that the point is the starting point of keystroke action;

[0038] Step 2.3.3, determine the end position of the keystroke by the start point of the keystroke, while setting an adjustment factor v; keystroke start point s p and end point e p The calculation formula is as follows:

[0039] s p = h p + v-w-p left (4)

[0040] e p = h p + v-w-t avg + p right (5)

[0041] Wherein, h p represents the position of p1point, w represents the length of the sliding window, t avg represents the average keystroke length; p left and p right represent the adjustment parameters for ensuring that the detected interval contains a complete keystroke waveform;

[0042] Step 2.3.4, fuse the keystroke positions detected by each antenna pair, take the detection result of the antenna pair composed of the 2ndtransmitting antenna and the 1streceiving antenna which is more sensitive to keystroke action as the main basis, and correct the detection results of other antenna pairs; Specifically, if the position of the keystroke detected by the antenna pair composed of the 2ndtransmitting antenna and the 1streceiving antenna is [p 21 l ,p 21 r ], then fuse the detection results of each antenna pair in the following way:

[0043]

[0044]

[0045] Wherein, i represents the antenna pair number, N represents the total number of antenna pairs, t avg represents the average keystroke duration, s represents the start position of the keystroke, and e represents the end position of the keystroke;

[0046] Step 2.4, keystroke action classification; the specific process is as follows:

[0047] Step 2.4.1, use the keystroke position detected in the previous step to extract the waveform when the keystroke action occurs from the original CSI data;

[0048] Step 2.4.2, Butterworth low-pass filtering and PCA processing are performed on the extracted waveform, and the first three principal components are taken;

[0049] Step 2.4.3, the principal components of each antenna pair are twice discrete wavelet transformed to extract features;

[0050] Step 2.4.4, the extracted features are sent into SVM to realize keystroke classification.

[0051] Further, the WiNum algorithm uses three anti-interference measures, as follows:

[0052] (1) After detecting a keystroke, jump back a distance for detection;

[0053] (2) The difference processing result of the data is processed by a sliding window;

[0054] (3) Set an interval of length L, when the processed waveform first exceeds the set threshold, observe whether the number of times that the waveform exceeds the threshold in the subsequent L length interval is greater than the set value, and judge whether a keystroke occurs according to the number of times.

[0055] The beneficial technical effects brought by the present application are:

[0056] The present application designs a data acquisition platform and a corresponding data processing algorithm WiNum of a keyboard keystroke recognition system based on WiFi signals. The CSI data collected by the data acquisition platform is processed by the designed keystroke recognition algorithm (WiNum) to infer whether a keystroke occurs and the category of the keystroke. The recognition accuracy and stability are high, and no other sensor is needed, which greatly reduces the development cost and has high portability. In addition, the system is easy to install and maintain, and can operate for a long time. Unlike other keyboard keystroke recognition systems, this system only needs WiFi network equipment to complete keyboard keystroke recognition, the hardware deployment is simple, and the system cost is greatly reduced. In addition, compared with previous WiFi-based keystroke recognition systems, this system has the advantages of high recognition accuracy and strong anti-interference performance. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 It is a CSI data acquisition hardware platform based on WiFi signal keystroke recognition in the embodiment of the present application;

[0058] Figure 2 It is a 15th subcarrier waveform graph of the CSI signal collected in the embodiment of the present application;

[0059] Figure 3 It is a feature waveform graph extracted by the WiNum algorithm in the embodiment of the present application, wherein (a) and (b) correspond to the feature waveform graphs extracted when the number keys 6 and 7 are struck, respectively.

[0060] Figure 4 This is a schematic diagram illustrating the keystroke misdetection problem solved by the anti-interference measures in this embodiment of the invention;

[0061] Figure 5 This is a diagram illustrating the keystroke position recognition effect in an embodiment of the present invention. Detailed Implementation

[0062] The basic idea of ​​this invention is:

[0063] This invention utilizes the principle that hand movements affect WiFi signal propagation when a person types on a keyboard. It designs a WiFi Channel State Information (CSI) data acquisition hardware platform and a keystroke recognition algorithm – WiNum. The CSI data acquisition platform collects CSI data carrying user keystroke information. The WiFi signal receiver is equipped with a trained WiNum neural network model, which processes the collected CSI data to achieve keystroke recognition. This system boasts high speed and stability, requires no additional sensors, reduces development costs, and is highly portable. Furthermore, the entire system is easy to install, maintain, and operate for extended periods.

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0065] like Figure 1 As shown, a keystroke recognition system based on WiFi signals includes a CSI data acquisition hardware platform and a keyboard. The data acquisition hardware platform mainly consists of a WiFi signal transmitter and a WiFi signal receiver, and is configured to acquire WiFi Channel State Information (CSI) data carrying keystroke action information. Wherein:

[0066] The WiFi signal transmitter can be a commercial wireless router or a laptop computer with three 5dB omnidirectional antennas connected to it to transmit WiFi signals.

[0067] The WiFi signal receiver is a desktop computer, which is equipped with a Linux system and an Intel 5300 or Atheros 9590 network card. It also has three external 5dB omnidirectional antennas for receiving data and recording CSI data containing keystroke information.

[0068] The antennas of the WiFi signal transmitter are placed in the designated positions. The three antennas are arranged vertically in a straight line and located to the right of the keyboard, numbered 0, 1, and 2 from bottom to top. The antennas are evenly distributed, spaced 40cm apart, with the closest point to the keyboard at 120cm, and 10cm above the keyboard.

[0069] The WiFi signal receiving end antenna is also placed according to the specified position, three antennas are placed around the left side of the keyboard, numbered 0, 1 and 2 from bottom to top, wherein the 0th antenna is 30 cm away from the keyboard, the 1st antenna is 20 cm away from the keyboard, and the 2nd antenna is 7 cm away from the keyboard. All the receiving antennas are bent to 120 degrees, so that the received CSI signal contains more keystroke action information.

[0070] The WiFi signal receiving end antenna and the WiFi signal sending end antenna are placed according to the specified position, on the one hand, so that the received CSI signal contains more keystroke action information, and on the other hand, so that the keystroke action is observed from different angles, thereby facilitating subsequent keystroke recognition.

[0071] From Figure 2 It can be seen that the CSI waveform collected when the keystroke action occurs has obvious changes. It is found through research that the CSI waveform changes caused by striking different keys are different, and the WiNum algorithm designed in the application is used to process the collected CSI data to infer the time and type of keystroke occurrence.

[0072] The keystroke recognition based on the WiFi signal in the application adopts the designed WiNum algorithm, the WiNum algorithm analyzes the law of the received and sent antennas at different positions on the collected keystroke waveform, and uses the law to design a fusion algorithm of the detection results of each antenna. The WiNum algorithm mainly consists of two parts: the first part is used to identify the position of the keystroke occurrence, and the second part is used to identify the type of the keystroke occurrence. The first part of the WiNum algorithm uses a double-scale sliding window to process data and uses a backward first-order difference to obtain the change trend of the data, and the second part uses a discrete wavelet transform (DWT) to extract features and uses a support vector machine to classify. The following will be described in detail.

[0073] The data processing process of the WiNum algorithm is as follows:

[0074] Step 1: pre-process the collected data to eliminate noise contained in part of the information; specifically including the following steps:

[0075] Step 1.1: because noise is generally high frequency, and signals containing keystroke action information are generally low frequency, a Butterworth low-pass filter is used to remove high-frequency interference, and the cut-off frequency w of the Butterworth low-pass filter is c , which can be obtained by the following formula:

[0076]

[0077] wherein f represents the highest frequency of the useful signal, and F represents the signal sampling frequency.

[0078] Step 1.2: Normalization is performed to facilitate subsequent calculations.

[0079] Step 1.3: PCA filtering is performed to take the first three principal components of the signal. This can compress the data while retaining a large amount of information, thereby reducing computational load and improving keystroke recognition speed.

[0080] Step 2: Keystroke action detection, which includes the following steps:

[0081] Step 2.1: Three receiving antennas and three transmitting antennas are combined in pairs to form nine antenna pairs. The principal component sequence of each antenna pair is processed using a sliding window. The purpose of sliding window processing is to suppress insignificant waveform changes. The larger the window, the more obvious the suppression effect. The calculation formula is as follows:

[0082]

[0083] where w represents the length of the sliding window, s i k represents the kth principal component sequence of the i antenna pair, j represents the position of the sequence midpoint, m i k represents the sliding window processing result of the kth principal component sequence of the i antenna pair.

[0084] Step 2.2: Add the sliding window processed principal component sequences of each antenna pair to form sequence S1.

[0085] Step 2.3: Perform first-order difference processing on S1 to obtain D1, which can reflect the trend of sequence S1. The calculation formula is as follows:

[0086] d i (j) = s i (j+1) - s i (j) (3)

[0087] where s i (j) represents the jth sample point of the i antenna pair, d i (j) represents the difference processing result of the j position of the i antenna pair.

[0088] Step 2.4: Set a threshold K. If D1 exceeds the threshold K for 7 times, it is considered that a keystroke has occurred, and the point is recorded as p0. Otherwise, the current point is not recorded.

[0089] Step 3: Keystroke position detection, which includes the following steps:

[0090] Step 3.1: Similar to obtaining S1, change the length of the sliding window to half of that used in processing S1 to obtain S2, which is used to find a more precise keystroke position.

[0091] Step 3.2: Starting from point p0, the position when sequence S2 increases or decreases 100 times consecutively is recorded as p1. This point is considered to be the starting point of the keystroke action.

[0092] Step 3.3: In practical applications, the duration of a keystroke generally fluctuates around a fixed value. Therefore, once the starting point of the keystroke is found, the ending point can be roughly determined. Additionally, considering that sliding window processing causes the detection results to shift forward, an adjustment factor v is needed. Keystroke starting point s p and the endpoint e p The calculation formula is as follows:

[0093] s p =h p +v·wp left (4)

[0094] e p =h p +v·w+t avg +p right (5)

[0095] Where w represents the length of the sliding window, t avg This indicates the average keystroke duration. left This indicates that the parameter to the left of the keystroke position is adjusted, p right This indicates that the parameters for adjusting the keystroke position are on the left.

[0096] Step 2.3.4: Merge the keystroke positions detected by each antenna pair, using the detection results of the antenna pair consisting of transmitting antenna 2 and receiving antenna 1, which is more sensitive to keystroke actions, as the primary basis, and correcting the detection results of other antenna pairs; specifically, let the keystroke position detected by the antenna pair consisting of transmitting antenna 2 and receiving antenna 1 be [p 21 l ,p 21 r The detection results of each antenna pair are then fused using the following formula:

[0097]

[0098]

[0099] in, i represents the antenna pair number, N represents the total number of antenna pairs, and t avg This represents the average keystroke duration, where 's' represents the start position of the keystroke and 'e' represents the end position of the keystroke.

[0100] Step 4: keystroke action classification, specifically including the following steps:

[0101] Step 4.1: using the keystroke position detected in the previous step, extract the waveform when the keystroke action occurs from the original CSI data.

[0102] Step 4.2: Butterworth low-pass filter and PCA processing are performed on the extracted waveform, and the first three principal components are taken.

[0103] Step 4.3: perform two discrete wavelet transforms on the principal components of each antenna pair to extract features, for example, Figure 3 Figures (a) and (b) show the extracted feature waveforms when the number keys 6 and 7 are struck, respectively.

[0104] Step 4.4: the extracted features are sent to the SVM to realize keystroke classification.

[0105] In addition, the WiNum algorithm also adds three anti-interference measures, which are as follows:

[0106] (1) After detecting a keystroke at any point in sequence D1, jump back a certain distance for detection, because after a keystroke occurs, a certain time must elapse before the next keystroke can be initiated, so the interference generated within the jumped distance will not affect the detection of the keystroke position.

[0107] (2) Since the D1 sequence is sensitive to noise, it is easy to cause keystroke false detection. Therefore, it is decided to perform sliding window processing on D1, which can suppress the large difference caused by noise and highlight the waveform changes caused by keystrokes.

[0108] (3) When the time interval between two consecutive keystrokes is large, noise is likely to cause the D1 sequence to exceed the set threshold multiple times within this interval, resulting in keystroke false detection, Figure 4 which shows this problem, where the points circled by the ellipse are false detection points. In order to solve the above problem, an interval of length L is set. When the D1 sequence exceeds the set threshold for the first time, observe whether the number of times the D1 sequence exceeds the threshold within the subsequent L length interval is greater than the set value, and determine whether a keystroke has occurred accordingly.

[0109] In this experiment, there are 9 keystroke categories, and 30 data are collected for each category. The experimental results show that the keystroke recognition accuracy of the system can reach more than 77%, and the average intersection over union (IoU) between the predicted keystroke position and the actual position is greater than 0.83. Figure 5 shows the keystroke position detection effect.

[0110] Compared with the similar system, the embodiment of the application has higher accuracy and stronger anti-interference ability.

[0111] Of course, the above description is only for the preferred embodiment of the application, and the application is not limited to the above-mentioned embodiments. It should be noted that any person skilled in the art, under the teaching of the present application, all equivalent replacements, obvious modifications, and all falling within the scope of the present application should be protected by the present application.

Claims

1. A keyboard keystroke recognition method based on WiFi network signals, characterized in that, A keyboard keystroke recognition system based on WiFi network signals is adopted. The keyboard keystroke recognition system includes a CSI data acquisition hardware platform and a keyboard. The CSI data acquisition hardware platform includes a WiFi signal transmitter and a WiFi signal receiver. Antennas connected to the WiFi signal receiver and antennas connected to the WiFi signal transmitter are respectively placed on the left and right sides of the keyboard. Among them, the WiFi signal transmitter is configured to transmit WiFi signals; WiFi signal receiver: configured to receive data and record CSI data containing keystroke information; The WiFi signal receiver is equipped with a trained WiNum neural network model. The WiNum neural network model uses the WiNum algorithm. The data collected by the CSI data acquisition hardware platform is processed by the WiNum algorithm to obtain the time of keystroke and the type of keystroke. The keyboard keystroke recognition method uses the WiNum algorithm to identify the location and type of keystrokes. The WiNum algorithm consists of two parts: the first part identifies the location of the keystroke, and the second part identifies the type of keystroke. The first part of the WiNum algorithm uses a dual-scale sliding window to process the data and backward first-order differencing to obtain the data's trend. The second part uses discrete wavelet transform for feature extraction and support vector machine for classification. The method specifically includes the following: Phase 1: Model Training Phase; the specific process is as follows: Step 1.1: Set up the CSI data acquisition hardware platform. After the platform is set up, the user presses the key to be recognized on the keyboard multiple times to collect the training data required by the WiNum neural network model. Step 1.2: Use the collected data to perform offline training and testing on the WiNum neural network model; Before training, the collected data is preprocessed to separate the required CSI amplitude information from the collected data, and the data is labeled and divided into training and test sets. Then, the training set is fed into a neural network model written according to the WiNum algorithm for multiple iterations to gradually bring the model parameters closer to the optimal. Step 1.3: After the model training is completed, test it with the test set and check the recognition effect. If the model recognition effect is good, output the model. If the recognition effect is not good, adjust the model parameters and retrain until the model recognition effect is good. Phase Two: Online Identification Phase; Step 2.1: Load the trained WiNum neural network model into the WiFi signal receiver; when the user types on the keyboard, collect the corresponding CSI data, and preprocess this data to remove noise from some of the information; the specific process is as follows: Step 2.1.1: Apply a Butterworth low-pass filter to the data to remove high-frequency interference. The cutoff frequency of the Butterworth low-pass filter is w. c It can be obtained from the following formula: Where f represents the highest frequency of the useful signal, and F represents the signal sampling frequency; Step 2.1.2: Perform normalization processing; Step 2.1.3: Perform PCA filtering and extract the first three principal components of the data signal; Step 2.2: Detect whether a keystroke action has occurred; the specific process is as follows: Step 2.2.1: The three receiving antennas and three transmitting antennas are paired to form nine antenna pairs. A sliding window process is applied to the principal component sequences of each antenna pair. The calculation formula is shown below: Where w represents the length of the sliding window, s i k Let m represent the k-th principal component sequence of the i-th antenna pair, j represent the position of a point in the sequence, and m represent the position of a point in the sequence. i k This represents the result of sliding window processing of the k-th principal component sequence of the i-th antenna pair; Step 2.2.2: Add the principal component sequences of each antenna pair after sliding window processing to form sequence S1; Step 2.2.3: Perform first-order difference processing on S1 to obtain D1, the calculation formula of which is shown below: d i (j)=s i (j+1)-s i (j) (3) where s i (j) represents the i-th antenna pair for the j-th sample point, d i (j) represents the differential processing result of antenna i at position j; Step 2.2.4: Set a threshold K. If D1 exceeds K a certain number of times, it is considered that a keystroke has occurred, and this point is recorded as point p0. Otherwise, the current point is not recorded. Step 2.3, Keystroke Position Detection; the specific process is as follows: Step 2.3.1: Similar to obtaining S1, change the length of the sliding window to half of that used when processing S1 to obtain S2; Step 2.3.2: Starting from point p0, the position of the sequence S2 when it continuously increases or decreases a certain number of times is recorded as p1. This point is considered to be the starting point of the keystroke action. Step 2.3.3: Determine the end position of the keystroke based on the start point of the keystroke, and simultaneously set an adjustment factor v; keystroke start point s p and the endpoint e p The calculation formula is as follows: s p =h p +v·w-p left (4) e p =h p +v·w+t avg +p right (5) Among them, h p Let p1 represent the position of point p1, w represent the length of the sliding window, and t represent the position of point p1. avg Indicates the average keystroke duration; p left and p right This refers to the adjustment parameters used to ensure that the detected interval contains the complete keystroke waveform; Step 2.3.4: Merge the keystroke positions detected by each antenna pair, using the detection results of the antenna pair consisting of transmitting antenna 2 and receiving antenna 1, which is more sensitive to keystroke actions, as the primary basis, and correcting the detection results of other antenna pairs; specifically, let the keystroke position detected by the antenna pair consisting of transmitting antenna 2 and receiving antenna 1 be [p 21 l ,p 21 r The detection results of each antenna pair are then fused using the following formula: in, i represents the antenna pair number, N represents the total number of antenna pairs, and t avg This represents the average keystroke duration, where 's' represents the start position of the keystroke and 'e' represents the end position of the keystroke. Step 2.4, Keystroke Action Classification; the specific process is as follows: Step 2.4.1: Using the keystroke position detected in the previous step, extract the waveform of the keystroke action from the raw CSI data; Step 2.4.2: Perform Butterworth low-pass filtering and PCA processing on the extracted waveform, and take the first three principal components; Step 2.4.3: Perform two discrete wavelet transforms on the principal components of each antenna pair to extract features; Step 2.4.4: Input the extracted features into an SVM to perform keystroke classification.

2. The keyboard keystroke recognition method based on WiFi network signals according to claim 1, characterized in that, The WiFi signal transmitter uses a laptop or commercial wireless router. The transmitter is connected to three antennas arranged vertically in a straight line. The antennas are evenly distributed, spaced 40cm apart, and placed 120cm from the right edge of the keyboard. The antennas are positioned 10cm above the keyboard.

3. The keyboard keystroke recognition method based on WiFi network signals according to claim 1, characterized in that, The WiFi signal receiver is a desktop computer equipped with a network card. The receiver is connected to three antennas, which are arranged around the keyboard and numbered 0, 1, and 2 from bottom to top. Antenna 0 is 30cm away from the keyboard, antenna 1 is 20cm away from the keyboard, and antenna 2 is 7cm away from the keyboard.

4. The keyboard keystroke recognition method based on WiFi network signals according to claim 1, characterized in that, The antenna is a 5dB omnidirectional antenna, with the three antennas connected to the receiver bent at 120 degrees.

5. The keyboard keystroke recognition method based on WiFi network signals according to claim 1, characterized in that, The WiNum algorithm employs three anti-interference measures, as follows: (1) After a keystroke is detected, jump back a certain distance to perform the detection; (2) Perform sliding window processing on the difference processing results of the data; (3) Set an interval of length L. When the processed waveform exceeds the set threshold for the first time, observe whether the number of times the waveform exceeds the threshold in the subsequent L-length interval is greater than the set value, and determine whether a keystroke has occurred.

Images