Keystroke identification method based on acoustic eavesdropping

Abstract

The invention discloses a keystroke identification method based on acoustic eavesdropping. The method comprises the following steps: S1, acquiring keystroke data; s2, performing pre-emphasis processing on the keystroke data, and removing background noise by using an adaptive spectral subtraction method to obtain a clean keystroke signal; s3, performing fast Fourier transform on the clean keystroke signal, calculating frequency spectrum energy, separating the keystroke signal into single keystroke segments, and ensuring that each segment only contains one keystroke behavior; s4, extracting spectrogram and Mel spectrogram features of each keystroke segment; s5, introducing a multi-scale channel attention module and an iterative attention feature fusion module, carrying out feature fusion on the spectrogram and Mel spectrogram features, and endowing key features with higher weights through an attention mechanism; s6, inputting the fused feature data into a convolutional neural network model for training, and using the model to classify and identify the keystroke behavior; according to the method, the accuracy and practicability of keystroke recognition are improved, and the method is more suitable for a complex acoustic environment.

Description

Technical Field

[0001] The invention belongs to the technical field of keyboard sound recognition of acoustic side channel attacks, and in particular relates to a keystroke recognition method based on acoustic eavesdropping. Background Art

[0002] The rapid development of information technology has made the Internet and smart devices an indispensable part of modern society. These devices are equipped with high-precision sensors, which greatly enrich the user's interactive experience and improve the convenience of life. However, the other side of this technological advancement is that it provides attackers with new ways to steal users' sensitive information. Among the many ways to interact with smart devices, keyboard input has become a major channel for information leakage due to its universality. Therefore, eavesdropping attacks on keyboard input content, especially side channel attacks, have become an important research topic in the field of information security. At present, such attacks are mainly divided into three types: motion signal attacks based on inertial sensors, attacks based on wireless communication signals, and attacks based on acoustic signals. Among these three attack methods, attacks based on acoustic signals have become the focus of researchers because they directly use the sound waves and vibration characteristics generated when the keyboard is tapped.

[0003] The effectiveness of keyboard side channel attacks based on acoustic signals is based on an in-depth analysis of the acoustic characteristics of keyboard tapping. Each keyboard tap generates a sound wave with unique acoustic characteristics, which not only contains information about the physical characteristics of the keys themselves, but also reflects the user's typing style and habits. An attacker can infer the user's key sequence by analyzing the characteristics of these sound waves, thereby obtaining sensitive information. It is worth noting that since users may use different fingers, strengths, or angles when tapping different keys, this will cause the sound produced by each tap to differ in intensity and timbre, and these differences provide additional clues for attackers. Since 2004, when researchers such as R. Agrawal experimentally verified the feasibility of using mechanical keyboard sound to recover input data, the research on acoustic side channel attack technology has received widespread attention in the field of information security and has gradually become a research hotspot in this field.

[0004] Although acoustic side-channel attack technology has broad application potential in theory, it faces many challenges and limitations in practical applications. First, the interference of environmental noise is a major problem. In actual application environments, there are usually various background noises, such as human voices and equipment noises, which will interfere with the collection of keystroke sounds and affect the accuracy of the attack. Traditional noise reduction methods often have difficulty in effectively extracting keystroke features in complex environments. Secondly, the efficiency and accuracy of acoustic feature extraction need to be improved. Existing methods are difficult to balance efficiency and accuracy when extracting features, especially the insufficient capture of high-frequency components and detail features, resulting in incomplete feature expression, which affects the effect of subsequent classification. In addition, existing technologies usually use a single feature extraction method, which fails to fully utilize the combined advantages of multiple features, and traditional classification models have limited performance when processing complex acoustic features. It is difficult to effectively identify subtle key position differences and individual user differences, which limits the generalization ability and recognition accuracy of the model. Finally, real-time performance and computational complexity are also important constraints in practical applications. Keystroke recognition technology needs to have strong real-time processing capabilities, but complex feature extraction and classification algorithms often have large computational workloads, resulting in slow system response speed and difficulty in meeting real-time requirements. In summary, the existing acoustic side-channel attack technology has obvious deficiencies in environmental adaptability, feature extraction efficiency, feature fusion strategy, model adaptability and real-time processing, which limits its effectiveness and feasibility in practical applications. Summary of the invention

[0005] The present invention aims to provide a keystroke recognition method based on acoustic eavesdropping, focusing on solving the challenges faced by the prior art in processing complex acoustic signals, especially in how to effectively fuse multiple acoustic features to achieve accurate keystroke classification tasks. The prior art usually fails to make full use of various acoustic features and lacks a complete feature fusion strategy, resulting in limited model capabilities in keystroke recognition. To this end, the present invention adopts advanced feature fusion technology and deep learning models to overcome these technical limitations, significantly improve the accuracy and practicality of keystroke recognition, and make it more suitable for complex acoustic environments.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A keystroke recognition method based on acoustic eavesdropping specifically comprises the following steps:

[0008] S1, obtaining keystroke data: using the mobile phone microphone to collect keystroke audio data and obtain the acoustic signal generated during the keystroke process;

[0009] S2, keystroke data preprocessing: pre-emphasize the keystroke data obtained in S1, enhance the high-frequency components of the signal, and use adaptive spectral subtraction to remove background noise to obtain a clean keystroke signal;

[0010] S3, keystroke event detection: perform fast Fourier transform on the keystroke signal preprocessed by S2, calculate the spectrum energy, separate the keystroke signal into individual keystroke segments, and ensure that each keystroke segment contains only one keystroke behavior;

[0011] S4, feature extraction: extract spectrogram and mel-spectrogram features for each keystroke segment, and extract acoustic features from the keystroke segment;

[0012] S5, feature fusion and attention mechanism: introduce multi-scale channel attention module and iterative attention feature fusion module to fuse the spectrogram and Mel-spectrogram features, and give higher weights to key features through the attention mechanism;

[0013] S6, inputting the fused feature data into the convolutional neural network model for training, using the model to classify and identify the keystroke behavior, and finally obtaining the keystroke recognition result.

[0014] Furthermore, the keystroke data in S1 is collected using a highly sensitive device in a controlled environment, and the keystroke data contains audio features generated when the user taps the keyboard.

[0015] Furthermore, the specific method of preprocessing the keystroke data in S2 is:

[0016] S2.1, pre-emphasizes the keystroke data obtained by S1, and its operation can be expressed as:

[0017] y[n]=x[n]-αx[n-1]

[0018] Wherein, x[n] represents the original signal sequence, y[n] is the signal sequence after pre-emphasis processing, and parameter α is the pre-emphasis coefficient;

[0019] S2.2, the result obtained by pre-emphasis in S2.1 is subjected to denoising. First, the average value of the first five frames of keystroke data is taken to construct a noise model, and the spectrum is averaged to obtain a noise spectrum representing the background noise, that is:

[0020]

[0021] Where n is the number of frames used to calculate the average value. is the average estimate of the noise spectrum, which is used to represent the noise intensity at frequency ω, and represents the frequency domain representation of the acoustic signal in the i-th frame;

[0022] In the spectral subtraction method, it is assumed that the noise interference in the original signal is additive, and the collected keystroke data can be regarded as a linear superposition of the keystroke signal and the noise signal. Therefore, by estimating the noise component and subtracting it from the keystroke data, a clean signal P that is closer to the original can be restored.N ,In order to further avoid the negative signal in spectral subtraction, the denoised signal is

[0023]

[0024] Among them, P N (ω) represents the signal spectrum after noise reduction processing, that is, the estimated value of the clean signal. At frequency ω, it is the result of removing the noise signal from the keystroke data. Y(ω) represents the spectrum of the original keystroke data, which contains the mixed spectrum of the keystroke signal and the noise signal. N is the subtraction factor calculated from the signal-to-noise ratio, represents the mean estimate of the noise spectrum, represents the final signal spectrum value after noise reduction, which is an estimate of the clean signal spectrum. N It is used to determine the adjustment coefficient to avoid negative numbers when the amplitude of the signal after noise reduction is less than the noise amplitude. In order to adaptively adjust the noise estimation model, the noise spectrum is also updated according to the signal-to-noise ratio (SNR) of the calculated signal. The calculation method of the signal-to-noise ratio is:

[0025]

[0026] When the signal-to-noise ratio in a frame is lower than the set empirical threshold, the noise estimation spectrum is adaptively adjusted according to the noise size. The update calculation process of the noise estimation model is as follows:

[0027]

[0028] in, represents the updated noise spectrum estimate, G N Represents a gain factor between 0 and 1 that controls the current noise estimate and the weight of the observed spectrum Y(ω) of the current frame in the noise estimation.

[0029] Furthermore, the specific approach of S3 is:

[0030] Perform a fast Fourier transform on the keystroke signal preprocessed by S2, convert the time domain into the frequency domain, and check the frequency domain of the keystroke signal. The calculation process is as follows:

[0031]

[0032] in, is the rotation factor, and its calculation formula is The keystroke signal is x(n), n=0,1,…,N-1;

[0033] Look at the unique characteristics of the keystroke signal frequency and calculate the spectral energy of the keystroke signal, Ek Indicates that the empirical threshold is used to determine whether to enter the keystroke process. When the E of any of the two channels k When the empirical threshold is exceeded, it is considered that the keystroke process has begun, and this time point is used as the starting point of the keystroke segment. The next signal of the empirical threshold length is used as a separate keystroke segment. After the key separation algorithm, multiple keystroke segments are obtained, and each keystroke segment contains only one keystroke behavior.

[0034] Furthermore, the specific steps of S4 are as follows:

[0035] Spectrogram feature extraction: Use the built-in spectrogram function in Matlab to calculate the short-time Fourier transform of the audio signal of the keystroke fragment to generate a spectrogram;

[0036] Mel-spectrogram feature extraction, divides the keystroke fragment into several frames, and the corresponding sample numbers of the frame length L and frame shift S are:

[0037] N=L×f s ,H=S×f s

[0038] Among them, f s is the signal sampling rate;

[0039] After the framing is completed, a Hanning window function is added to each frame, and then a fast Fourier transform is performed on each frame after framing and windowing. The magnitude of the FFT transform result is taken, squared and divided by the corresponding number of FFT points to obtain the power spectrum. The power spectrum is then converted to the Mel frequency scale through the Mel filter group. The corresponding frequency is linear on the Mel frequency scale. The last step is to logarithmically transform each element of the Mel spectrum graph to obtain the Mel spectrum graph.

[0040] Furthermore, the specific approach of S5 is:

[0041] Multi-scale channel attention module MS-CAM, the size of the input feature X is C×H×W, which is compressed into a C×1×1 tensor through global average pooling to extract the global statistical information of each channel; then, the channel is compressed through a point convolution to obtain a dimension of The features are processed by batch normalization and ReLU activation function to increase nonlinearity. Then, the dimension of the features is restored back to C×1×1 through the second point convolution and then processed by batch normalization again.

[0042] The restored features are fused with the input features, and the Sigmoid activation function is applied through the addition operation to generate the channel attention weight, whose size is C×H×W; finally, the input feature X is multiplied by the generated channel attention weight to obtain the output feature X′; the two branches on the right represent the channel attention of the global feature and the channel attention of the local feature respectively. The channel attention calculation formula L(X) of the local feature is as follows:

[0043] L(X)=B(PWConv2(δ(B(PWConv1(X)))))

[0044] Among them, PWConv represents the point convolution operation, B represents batch normalization, and δ represents the ReLU activation function; this module can generate the attention weights of a specific channel by combining global statistical information and point convolution adjustment;

[0045] The multi-scale channel attention module MS-CAM is applied to the feature fusion of keystroke recognition. The specific approach is as follows:

[0046] First, by adjusting the multi-scale pooling and convolution structure, the characteristic differences of keystroke sounds in different frequency ranges are highlighted;

[0047] Secondly, the fusion of global timing features and local detail features in the keystroke signal is added to ensure that the convolutional neural network model can not only pay attention to the overall timing features of the keystroke fragment, but also capture the local change detail features in the keystroke fragment; an adaptive initial feature fusion strategy is introduced to make the initial fusion result more in line with the characteristics of the audio data; in addition, the iterative attention mechanism iAFF is adopted. iAFF uses the MS-CAM module for feature enhancement and fusion, and refines the feature fusion of the keystroke fragment signal after multiple iterations.

[0048] Furthermore, the specific approach of S6 is:

[0049] Convolutional neural network (CNN) is used for deep learning model training. CNN consists of two convolutional layers and one fully connected layer. Each convolutional layer is followed by a ReLU activation function. At the same time, the size of the feature map is reduced through the pooling layer to reduce the computational complexity and retain key features. During the deep learning model training, the cross entropy loss function is used to optimize the recognition results. After training, the deep learning model uses the Softmax function to generate recognition results in the output layer, and selects the category with the highest probability as the final prediction result to complete the keystroke recognition task.

[0050] A computer terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, a keystroke recognition method based on acoustic eavesdropping as described above is implemented.

[0051] A computer-readable medium stores a computer program, which is executed by a processor to implement a keystroke recognition method based on acoustic eavesdropping as described above.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention extracts features from keystroke signals and effectively captures the keystroke signals. By implementing a feature fusion strategy and combining the attention mechanism, a higher weight is given to certain features during feature fusion, thus achieving a deeper understanding of the keystroke signals. Experimental results show that after feature fusion, the model is superior to a single feature that has not undergone feature fusion in terms of performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0055] Figure 2 is a schematic diagram of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0056] Figure 3 It is a Mel-map feature extraction flow chart of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0057] Figure 4 It is a schematic diagram of the structure of a multi-scale channel attention module of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0058] Figure 5 It is a schematic diagram of the structure of an iterative attention feature fusion module of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0059] Figure 6 It is a schematic diagram of the performance accuracy and loss rate of a CNN model of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0060] Figure 7 It is a frequency domain distribution diagram of a keystroke recognition method based on acoustic eavesdropping of the present invention. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings and specific implementation methods. Figure 1-2 As shown, a keystroke recognition method based on acoustic eavesdropping specifically includes the following steps:

[0062] S1, obtain keystroke data: obtain audio data from Harrison et al. in the document "Harrison J, Toreini E, Mehrnezhad MA practical deep learning-based acoustic side channel attack on keyboards. 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 2023: 270-280."; these data are collected in a controlled environment using highly sensitive equipment, reflecting the audio features generated when the user taps the keyboard, providing a reliable data basis for subsequent keystroke recognition;

[0063] S2, preprocesses the keystroke data obtained by S1;

[0064] S2.1, pre-emphasis, first, the keystroke data obtained by S1 is pre-emphasized. Pre-emphasis is an effective signal compensation technology, the main purpose of which is to enhance the high-frequency components of the signal. Its operation can be expressed as:

[0065] y[n]=x[n]-αx[n-1]

[0066] Wherein, x[n] represents the original signal sequence, y[n] is the signal sequence after pre-emphasis processing, and parameter α is the pre-emphasis coefficient;

[0067] S2.2, denoising, denoising the result obtained by pre-emphasis in S2.1, first take the average value of the first 5 frames of keystroke data to build a noise model, and average its spectrum to obtain the noise spectrum representing the background noise, that is:

[0068]

[0069] Where n is the number of frames used to calculate the average value. is the average estimate of the noise spectrum, which is used to represent the noise intensity at frequency ω, and represents the frequency domain representation of the acoustic signal in the i-th frame;

[0070] In the spectral subtraction method, it is assumed that the noise interference in the original signal is additive, and the collected keystroke data can be regarded as a linear superposition of the keystroke signal and the noise signal. Therefore, by estimating the noise component and subtracting it from the keystroke data, a clean signal P that is closer to the original can be restored. N ,In order to further avoid the negative signal in spectral subtraction, the denoised signal is

[0071]

[0072] Among them, P N (ω) represents the signal spectrum after noise reduction processing, that is, the estimated value of the clean signal. At frequency ω, it is the result of removing the noise signal from the keystroke data. Y(ω) represents the spectrum of the original keystroke data, which contains the mixed spectrum of the keystroke signal and the noise signal. N is the subtraction factor calculated from the signal-to-noise ratio, represents the mean estimate of the noise spectrum, represents the final signal spectrum value after noise reduction, which is an estimate of the clean signal spectrum. N It is used to determine the adjustment coefficient to avoid negative numbers when the amplitude of the signal after noise reduction is less than the noise amplitude. In order to adaptively adjust the noise estimation model, the noise spectrum is also updated according to the signal-to-noise ratio (SNR) of the calculated signal. The calculation method of the signal-to-noise ratio is:

[0073]

[0074] When the signal-to-noise ratio in a frame is lower than the set empirical threshold, the noise estimation spectrum is adaptively adjusted according to the noise size. The update calculation process of the noise estimation model is as follows:

[0075]

[0076] in, represents the updated noise spectrum estimate, G N Represents a gain factor between 0 and 1 that controls the current noise estimate and the weight of the observed spectrum Y(ω) of the current frame in the noise estimation;

[0077] S3, keystroke event detection: Perform fast Fourier transform on the keystroke signal preprocessed by S2, convert the time domain into the frequency domain, and check the frequency domain of the keystroke signal. The calculation process is as follows:

[0078]

[0079] in, is the rotation factor, and its calculation formula is The keystroke signal is x(n), n=0,1,…,N-1;

[0080] Look at the unique characteristics of the keystroke signal frequency and calculate the spectral energy of the keystroke signal, E k Indicates that the empirical threshold is used to determine whether to enter the keystroke process. When the E of any of the two channels kWhen the empirical threshold is exceeded, it is considered that the keystroke process has begun, and this time point is taken as the starting point of the keystroke segment. The next signal of the empirical threshold length is taken as a separate keystroke segment. After the key separation algorithm, multiple keystroke segments are obtained, and each keystroke segment contains only one keystroke behavior.

[0081] S4, feature extraction: acoustic features are extracted from the keystroke fragments obtained in S3, including spectrogram and mel-spectrogram features;

[0082] Spectrogram feature extraction: Use the built-in spectrogram function in Matlab to calculate the short-time Fourier transform of the audio signal of the keystroke fragment to generate a spectrogram;

[0083] Mel-spectrogram feature extraction, divides the keystroke fragment into several frames, and the corresponding sample numbers of the frame length L and frame shift S are:

[0084] N=L×f s ,H=S×f s

[0085] Among them, f s is the signal sampling rate;

[0086] After the framing is completed, a Hanning window function is added to each frame, and then a fast Fourier transform is performed on each frame after framing and windowing. The magnitude of the FFT transform result is taken, squared, and then divided by the corresponding number of FFT points to obtain the power spectrum. The power spectrum is then converted to the Mel frequency scale through the Mel filter bank. The corresponding frequency is linear on the Mel frequency scale. The last step is to logarithmically transform each element of the Mel spectrum graph to obtain the Mel spectrum graph. The algorithm flow chart is as follows Figure 3 As shown;

[0087] S5, feature fusion and attention mechanism: introduce multi-scale channel attention module and iterative attention feature fusion module to fuse the spectrogram and Mel-spectrogram features, and use the attention mechanism to strengthen the weight of key features such as the sound generated by fingers pressing the keyboard.

[0088] The structure diagram of the multi-scale channel attention module MS-CAM is as follows Figure 4 As shown in Figure 1, the size of the input feature X is C×H×W, which is compressed into a C×1×1 tensor through global average pooling to extract the global statistical information of each channel; next, the channel is compressed through a point convolution to obtain a dimension of The features are processed by batch normalization and ReLU activation function to increase nonlinearity. Then, the dimension of the features is restored back to C×1×1 through the second point convolution and then processed by batch normalization again.

[0089] The restored features are fused with the input features, and the Sigmoid activation function is applied through the addition operation to generate the channel attention weight, whose size is C×H×W; finally, the input feature X is multiplied by the generated channel attention weight to obtain the output feature X′; the two branches on the right represent the channel attention of the global feature and the channel attention of the local feature respectively. The channel attention calculation formula L(X) of the local feature is as follows:

[0090] L(X)=B(PWConv2(δ(B(PWConv1(X)))))

[0091] Among them, PWConv represents the point convolution operation, B represents batch normalization, and δ represents the ReLU activation function; this module can generate attention weights for specific channels by combining global statistical information and point convolution adjustment, helping the model to better focus on important channel information and improve the expressiveness of features.

[0092] The multi-scale channel attention module MS-CAM is applied to the feature fusion of keystroke recognition. The specific approach is as follows:

[0093] First, by adjusting the multi-scale pooling and convolution structure, the characteristic differences of keystroke sounds in different frequency ranges are highlighted, so that the convolutional neural network model in S6 can more effectively capture different frequency and timing features;

[0094] Secondly, the fusion of global timing features and local detail features in the keystroke signal is added to ensure that the convolutional neural network model can not only focus on the overall timing features of the keystroke fragment, but also capture the local change detail features in the keystroke fragment; an adaptive initial feature fusion strategy is introduced to make the initial fusion result more in line with the characteristics of the audio data, thereby providing higher quality input for the attention module; in addition, an iterative attention mechanism iAFF is adopted, such as Figure 5 As shown in the figure, iAFF uses the MS-CAM module twice for feature enhancement and fusion, and refines the feature fusion of the keystroke segment signal through multiple iterations, making it clearer and more comprehensive in expressing the timing features and local features of the keystroke segment, thereby improving the model's robustness in dealing with dynamic noise changes and further enhancing the accuracy and stability of keystroke recognition.

[0095] S6, deep learning model training and recognition output: A convolutional neural network (CNN) is used for deep learning model training. CNN consists of two convolutional layers and one fully connected layer. Each convolutional layer is followed by a ReLU activation function to enhance the model's ability to express nonlinear features. At the same time, the size of the feature map is reduced through the pooling layer to reduce computational complexity and retain key features. During the deep learning model training process, the cross-entropy loss function is used to optimize the recognition results to improve the classification accuracy of the deep learning model for keystroke data. After the training is completed, the deep learning model uses the Softmax function to generate recognition results in the output layer and selects the category with the highest probability as the final prediction result to complete the keystroke recognition task.

[0096] The present invention extracts spectrogram features and Mel-map features from keystroke signals, effectively capturing the detailed features and global features of keystroke sounds. Furthermore, by implementing a feature fusion strategy and combining an attention mechanism, a higher weight is given to certain features during feature fusion, which helps improve the recognition accuracy of the model and achieves a deeper understanding of keystroke behavior. This strategy not only enhances the model's ability to recognize subtle differences in keystroke signals, but also improves generalization performance in complex environments.

[0097] Experiments were conducted using the audio data mentioned by Harrison et al. in the document "Harrison J, Toreini E, Mehrnezhad MA practical deep learning-based acoustic side channel attack on keyboards. 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 2023: 270-280." The experimental results show that the keyboard keystroke recognition method of the present invention achieved accuracies of 87% and 89% respectively by training using spectrogram features and Mel-spectrogram features alone. This result verifies the effectiveness of single feature extraction technology in keyboard keystroke recognition. However, when the feature fusion strategy is adopted, especially when combined with the attention mechanism, the accuracy of the model is improved to 91.54%, as shown in Figure 2. Figure 6 As shown in the figure, it demonstrates the significant effect of feature fusion combined with attention mechanism in improving recognition accuracy.

[0098] In terms of data preprocessing, adaptive spectral subtraction is used for noise reduction, and the signal-to-noise ratio is set to 5. The noise model is adaptively adjusted according to the signal-to-noise ratio. Keystroke separation is performed based on spectral energy. Figure 7The spectrum diagram shows that the keystroke signal has unique frequency domain characteristics in the range of 1-1000 Hz, so the accumulated value of the spectrum energy in this range is calculated, and the empirical threshold is set to 15. k When it exceeds 15, it is considered that the keystroke signal has entered the contact wave peak, and the next 180ms is used as the keystroke segment to separate the keystroke from other invalid sounds, thereby obtaining the keystroke segment.

[0099] In terms of feature extraction, in the spectrogram feature extraction parameters, nfft is set to 512, frame length is 18ms, window shift is 9ms, window function is Hanning window, and spectral features in the range of 0-8000hz are calculated. In the Mel map feature extraction parameters, the Mel filter is set to 40, and spectral features in the range of 0-3000hz are calculated. nfft, frame length and window shift are the same as the spectrogram feature parameters. When fusion of features, Mel map is used as feature X, and spectrogram is used as feature Y with a larger receptive field.

[0100] In terms of data set partitioning, the present invention uses 20 keystrokes of each 25 keystrokes for model training, and the remaining 5 keystrokes for verification and testing the performance of the model. The data was preprocessed and feature extracted on Matlab R2023b, and the model was trained on PyCharm Community Edition 2020.2.1.

[0101] During the training process, the data size of each batch of the model is set to 32, the learning rate is set to 0.001, and the cross-entropy loss function is used to measure the difference between the true label and the predicted label, and the model parameters are updated based on this combined with the adam optimizer.

[0102] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A keystroke recognition method based on acoustic eavesdropping, characterized in that: The specific steps include: S1, obtaining keystroke data: using the mobile phone microphone to collect keystroke audio data and obtain the acoustic signal generated during the keystroke process; S2, keystroke data preprocessing: pre-emphasize the keystroke data obtained in S1, enhance the high-frequency components of the signal, and use adaptive spectral subtraction to remove background noise to obtain a clean keystroke signal; S3, keystroke event detection: perform fast Fourier transform on the keystroke signal preprocessed by S2, calculate the spectrum energy, separate the keystroke signal into individual keystroke segments, and ensure that each keystroke segment contains only one keystroke behavior; S4, feature extraction: extract spectrogram and mel-spectrogram features for each keystroke segment, and extract acoustic features from the keystroke segment; S5, feature fusion and attention mechanism: introduce multi-scale channel attention module and iterative attention feature fusion module to fuse the spectrogram and Mel-spectrogram features, and give higher weights to key features through the attention mechanism; S6, inputting the fused feature data into the convolutional neural network model for training, using the model to classify and identify the keystroke behavior, and finally obtaining the keystroke recognition result.

2. A keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The keystroke data in S1 is collected in a controlled environment using highly sensitive equipment. The keystroke data contains the audio features generated when the user taps the keyboard.

3. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific method of preprocessing keystroke data in S2 is: S2.1, pre-emphasizes the keystroke data obtained by S1, and its operation can be expressed as: y[n]=x[n]-αx[n-1] Wherein, x[n] represents the original signal sequence, y[n] is the signal sequence after pre-emphasis processing, and parameter α is the pre-emphasis coefficient; S2.2, the result obtained by pre-emphasis in S2.1 is subjected to denoising. First, the average value of the first five frames of keystroke data is taken to construct a noise model, and the spectrum is averaged to obtain a noise spectrum representing the background noise, that is: Where n is the number of frames used to calculate the average value. is the average estimate of the noise spectrum, which is used to represent the noise intensity at frequency ω, and represents the frequency domain representation of the acoustic signal in the i-th frame; In the spectral subtraction method, it is assumed that the noise interference in the original signal is additive, and the collected keystroke data can be regarded as a linear superposition of the keystroke signal and the noise signal. Therefore, by estimating the noise component and subtracting it from the keystroke data, a clean signal P that is closer to the original can be restored. N ,In order to further avoid the negative signal in spectral subtraction, the denoised signal is Among them, P N (ω) represents the signal spectrum after noise reduction processing, that is, the estimated value of the clean signal. At frequency ω, it is the result of removing the noise signal from the keystroke data. Y(ω) represents the spectrum of the original keystroke data, which contains the mixed spectrum of the keystroke signal and the noise signal. N is the subtraction factor calculated from the signal-to-noise ratio, represents the mean estimate of the noise spectrum, represents the final signal spectrum value after noise reduction, which is an estimate of the clean signal spectrum. N It is used to determine the adjustment coefficient to avoid negative numbers when the amplitude of the signal after noise reduction is less than the noise amplitude. In order to adaptively adjust the noise estimation model, the noise spectrum is also updated according to the signal-to-noise ratio (SNR) of the calculated signal. The calculation method of the signal-to-noise ratio is: When the signal-to-noise ratio in a frame is lower than the set empirical threshold, the noise estimation spectrum is adaptively adjusted according to the noise size. The update calculation process of the noise estimation model is as follows: in, represents the updated noise spectrum estimate, G N Represents a gain factor between 0 and 1 that controls the current noise estimate and the weight of the observed spectrum Y(ω) of the current frame in the noise estimation.

4. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S3 is: Perform a fast Fourier transform on the keystroke signal preprocessed by S2, convert the time domain into the frequency domain, and check the frequency domain of the keystroke signal. The calculation process is as follows: in, is the rotation factor, and its calculation formula is The keystroke signal is x(n), n=0,1,…,N-1; Look at the unique characteristics of the keystroke signal frequency and calculate the spectral energy of the keystroke signal, E k Indicates that the empirical threshold is used to determine whether to enter the keystroke process. When the E of any of the two channels k When the empirical threshold is exceeded, it is considered that the keystroke process has begun, and this time point is used as the starting point of the keystroke segment. The next signal of the empirical threshold length is used as a separate keystroke segment. After the key separation algorithm, multiple keystroke segments are obtained, and each keystroke segment contains only one keystroke behavior.

5. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S4 is: Spectrogram feature extraction: Use the built-in spectrogram function in Matlab to calculate the short-time Fourier transform of the audio signal of the keystroke fragment to generate a spectrogram; Mel-spectrogram feature extraction, divides the keystroke fragment into several frames, and the corresponding sample numbers of the frame length L and frame shift S are: N=L×f s ,H=S×f s Among them, f s is the signal sampling rate; After the framing is completed, a Hanning window function is added to each frame, and then a fast Fourier transform is performed on each frame after framing and windowing. The magnitude of the FFT transform result is taken, squared and divided by the corresponding number of FFT points to obtain the power spectrum. The power spectrum is then converted to the Mel frequency scale through the Mel filter group. The corresponding frequency is linear on the Mel frequency scale. The last step is to logarithmically transform each element of the Mel spectrum graph to obtain the Mel spectrum graph.

6. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S5 is: Multi-scale channel attention module MS-CAM, the input feature X has a size of C×H×W, which is compressed into a C×1×1 tensor after global average pooling to extract global statistical information of each channel; Next, the channel is compressed through a point convolution to obtain a dimension of The features are processed by batch normalization and ReLU activation function to increase nonlinearity. Then, the dimension of the features is restored back to C×1×1 through the second point convolution and then processed by batch normalization again. The recovered features are fused with the input features, and the Sigmoid activation function is applied through the addition operation to generate the channel attention weight, whose size is C×H×W; finally, the input feature X is multiplied by the generated channel attention weight to obtain the output feature X′; The two branches on the right represent the channel attention of global features and the channel attention of local features respectively. The channel attention calculation formula L(X) of local features is as follows: L(X)=B(PWConv2(δ(B(PWConv1(X))))) Among them, PWConv represents the point convolution operation, B represents batch normalization, and δ represents the ReLU activation function; this module can generate the attention weights of a specific channel by combining global statistical information and point convolution adjustment; The multi-scale channel attention module MS-CAM is applied to the feature fusion of keystroke recognition. The specific approach is as follows: First, by adjusting the multi-scale pooling and convolution structure, the characteristic differences of keystroke sounds in different frequency ranges are highlighted; Secondly, the fusion of global timing features and local detail features in the keystroke signal is added to ensure that the convolutional neural network model can not only pay attention to the overall timing features of the keystroke fragment, but also capture the local change detail features in the keystroke fragment; an adaptive initial feature fusion strategy is introduced to make the initial fusion result more in line with the characteristics of the audio data; in addition, the iterative attention mechanism iAFF is adopted. iAFF uses the MS-CAM module for feature enhancement and fusion, and refines the feature fusion of the keystroke fragment signal after multiple iterations.

7. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S6 is: The convolutional neural network (CNN) is used for deep learning model training. CNN consists of two convolutional layers and one fully connected layer. Each convolutional layer is followed by a ReLU activation function. At the same time, the size of the feature map is reduced through the pooling layer to reduce the computational complexity and retain key features. During the deep learning model training process, the cross entropy loss function is used to optimize the recognition results. After training, the deep learning model uses the Softmax function to generate recognition results in the output layer and selects the category with the highest probability as the final prediction result to complete the keystroke recognition task.

8. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the keystroke recognition method based on acoustic eavesdropping as described in any one of claims 1 to 7 is implemented.

9. A computer readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement a keystroke recognition method based on acoustic eavesdropping as described in any one of claims 1 to 7.

Images