A keystroke recognition method based on acoustic eavesdropping

Abstract

The present invention discloses a keystroke recognition method based on acoustic eavesdropping, comprising: S1, collecting keystroke data; S2, performing pre-emphasis processing on the keystroke data, using adaptive spectral subtraction to remove background noise, and obtaining a clean keystroke signal; S3, performing fast Fourier transform on the clean keystroke signal, calculating spectral energy, and separating the keystroke signal into individual keystroke segments, ensuring that each segment contains only one keystroke behavior; S4, extracting spectrogram and Mel-spectrogram features for each keystroke segment; S5, introducing a multi-scale channel attention module and an iterative attention feature fusion module, performing feature fusion on the spectrogram and Mel-spectrogram features, and assigning higher weights to key features through the attention mechanism; S6, inputting the fused feature data into a convolutional neural network model for training, and using the model to classify and recognize keystroke behaviors; the method improves the accuracy and practicality of keystroke recognition, making it more suitable for complex acoustic environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of keyboard sound recognition of acoustic side-channel attacks, and in particular relates to a keystroke recognition method based on acoustic eavesdropping. Background Art

[0002] The rapid advancement of information technology has made networks and smart devices an integral part of modern society. These devices, equipped with high-precision sensors, have greatly enriched user experiences and improved convenience. However, this technological advancement also presents new avenues for attackers to steal sensitive user information. Among the many ways to interact with smart devices, keyboard input, due to its ubiquity, has become a major channel for information leakage. Therefore, eavesdropping attacks targeting keyboard input, particularly side-channel attacks, have become a significant research topic in the field of information security. Currently, these attacks are primarily categorized into three types: motion signal attacks based on inertial sensors, attacks based on wireless communication signals, and attacks based on acoustic signals. Of these three attack methods, acoustic signal-based attacks have attracted significant research attention because they directly exploit the sound waves and vibration characteristics generated by keyboard tapping.

[0003] The effectiveness of keyboard side-channel attacks based on acoustic signals relies on in-depth analysis of the acoustic characteristics of keyboard strokes. Each keystroke generates a sound wave with unique acoustic characteristics. These sound waves not only contain information about the physical properties of the keys themselves, but also reflect the user's typing style and habits. Attackers can analyze the characteristics of these sound waves to infer the user's keystroke sequence and thereby obtain sensitive information. It is worth noting that because users may use different fingers, force, or angles when striking different keys, the sound produced by each keystroke varies in intensity and timbre, providing additional clues for attackers. Since 2004, when researchers such as R. Agrawal experimentally demonstrated the feasibility of recovering input data from the sound of mechanical keyboards, research on acoustic side-channel attack techniques has garnered widespread attention in the information security field and has gradually become a research hotspot in the field.

[0004] While acoustic side-channel attacks hold broad theoretical potential, they face numerous challenges and limitations in practical applications. First, environmental noise interference is a major concern. Real-world environments often contain various background noises, such as human voices and equipment noise. These noises can interfere with the acquisition of keystroke sounds, impacting the accuracy of the attack. Traditional noise reduction methods often struggle to effectively extract keystroke features in complex environments. Second, the efficiency and accuracy of acoustic feature extraction need to be improved. Existing methods struggle to achieve both efficiency and accuracy when extracting features. In particular, they fail to adequately capture high-frequency components and detailed features, resulting in incomplete feature representation and hindering subsequent classification. Furthermore, existing technologies typically employ a single feature extraction approach, failing to fully leverage the combined advantages of multiple features. Traditional classification models also have limited performance when processing complex acoustic features, making it difficult to effectively identify subtle key position differences and individual user variations, limiting the model's generalization and recognition accuracy. Finally, real-time performance and computational complexity are also significant constraints in practical applications. Keystroke recognition technology requires strong real-time processing capabilities, but complex feature extraction and classification algorithms are often computationally intensive, resulting in slow system response and difficulty meeting real-time requirements. In summary, existing acoustic side-channel attack technologies have obvious deficiencies in environmental adaptability, feature extraction efficiency, feature fusion strategy, model adaptability, and real-time processing, which limits their effectiveness and feasibility in practical applications. Summary of the Invention

[0005] The present invention aims to provide a keystroke recognition method based on acoustic eavesdropping, focusing on solving the challenges faced by existing technologies when processing complex acoustic signals, especially in how to effectively fuse multiple acoustic features to achieve accurate keystroke classification tasks. Existing technologies generally fail to fully utilize various acoustic features and lack a complete feature fusion strategy, resulting in limited capabilities of the model in keystroke recognition. To this end, the present invention adopts advanced feature fusion technology and deep learning models to overcome these technical limitations, significantly improving the accuracy and practicality of keystroke recognition, making it more suitable for complex acoustic environments.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A keystroke recognition method based on acoustic eavesdropping specifically comprises the following steps:

[0008] S1, obtaining keystroke data: using the mobile phone microphone to collect keystroke audio data and obtain the acoustic signal generated during the keystroke process;

[0009] S2, keystroke data preprocessing: pre-emphasize the keystroke data obtained in S1 to enhance the high-frequency components of the signal, and use adaptive spectral subtraction to remove background noise to obtain a clean keystroke signal;

[0010] S3, keystroke event detection: Perform fast Fourier transform on the keystroke signal preprocessed by S2, calculate the spectrum energy, and separate the keystroke signal into individual keystroke segments, ensuring that each keystroke segment contains only one keystroke behavior;

[0011] S4, feature extraction: extract spectrogram and mel-spectrogram features for each keystroke segment, and extract acoustic features from the keystroke segment;

[0012] S5, Feature Fusion and Attention Mechanism: Introducing a multi-scale channel attention module and an iterative attention feature fusion module to fuse spectrogram and mel-spectrogram features, and giving higher weights to key features through the attention mechanism;

[0013] S6, inputs the fused feature data into the convolutional neural network model for training, uses the model to classify and identify the keystroke behavior, and finally obtains the keystroke recognition result.

[0014] Furthermore, the keystroke data in S1 is collected using a highly sensitive device in a controlled environment, and the keystroke data contains audio features generated when the user taps the keyboard.

[0015] Furthermore, the specific method of pre-processing the keystroke data in S2 is as follows:

[0016] S2.1 performs pre-emphasis processing on the keystroke data obtained by S1. The operation can be expressed as follows:

[0017] y[n]=x[n]-αx[n-1]

[0018] Where x[n] represents the original signal sequence, y[n] is the signal sequence after pre-emphasis processing, and the parameter α is the pre-emphasis coefficient;

[0019] S2.2, the result obtained by pre-emphasis in S2.1 is subjected to noise reduction. First, the average value of the first five frames of keystroke data is taken to construct a noise model. The spectrum is averaged to obtain the noise spectrum representing the background noise, that is:

[0020]

[0021] Where n is the number of frames used to calculate the average value, is the average value estimate of the noise spectrum, which is used to represent the noise intensity at frequency ω, and represents the frequency domain representation of the acoustic signal in the i-th frame;

[0022] In spectral subtraction, assuming that the noise interference in the original signal is additive, the collected keystroke data can be regarded as a linear superposition of the keystroke signal and the noise signal. Thus, by estimating the noise component and subtracting it from the keystroke data, a clean signal P that is closer to the original can be restored.N ,In order to further avoid the negative signal in spectrum subtraction, the denoised signal is

[0023]

[0024] Among them, P N (ω) represents the signal spectrum after noise reduction processing, that is, the estimated value of the clean signal. At frequency ω, it is the result of removing the noise signal from the keystroke data. Y(ω) represents the spectrum of the original keystroke data, which contains the mixed spectrum of the keystroke signal and the noise signal. N is the subtraction factor calculated based on the signal-to-noise ratio, represents the mean value estimate of the noise spectrum, Represents the final signal spectrum value after noise reduction, which is an estimate of the clean signal spectrum, β N It is used to determine the adjustment coefficient to avoid negative numbers when the amplitude of the signal after noise reduction is less than the noise amplitude. In order to adaptively adjust the noise estimation model, the noise spectrum is also updated according to the signal-to-noise ratio (SNR) of the calculated signal. The signal-to-noise ratio is calculated as follows:

[0025]

[0026] When the signal-to-noise ratio in a frame is lower than the set empirical threshold, the noise estimation spectrum is adaptively adjusted according to the noise level. The update calculation process of the noise estimation model is as follows:

[0027]

[0028] in, Represents the updated noise spectrum estimate, G N Represents a gain factor between 0 and 1, used to control the current noise estimate and the weight of the observed spectrum Y(ω) of the current frame in the noise estimation.

[0029] Furthermore, the specific approach of S3 is:

[0030] Perform a fast Fourier transform on the keystroke signal preprocessed by S2 to convert the time domain into the frequency domain. Check the frequency domain of the keystroke signal. The calculation process is as follows:

[0031]

[0032] in, is the rotation factor, and its calculation formula is The keystroke signal is x(n), n=0,1,…,N-1;

[0033] Look at the unique characteristics of the keystroke signal frequency and calculate the spectral energy of the keystroke signal, using Ek Indicates that the empirical threshold is used to determine whether to enter the keystroke process. When the E of any one of the two channels k When the empirical threshold is exceeded, it is considered that the keystroke process has begun, and this time point is used as the starting point of the keystroke segment. The signal of the following empirical threshold length is used as a separate keystroke segment. After the key separation algorithm, multiple keystroke segments are obtained, and each keystroke segment contains only one keystroke behavior.

[0034] Furthermore, the specific approach of S4 is:

[0035] Spectrogram feature extraction: Use the built-in spectrogram function in Matlab to calculate the short-time Fourier transform of the audio signal of the keystroke fragment to generate a spectrogram;

[0036] Mel-spectrogram feature extraction divides the keystroke segment into several frames, and the corresponding sample numbers of the frame length L and frame shift S are:

[0037] N=L×f s ,H=S×f s

[0038] Among them, f s is the signal sampling rate;

[0039] After the framing is completed, a Hanning window function is added to each frame, and then a fast Fourier transform is performed on each frame after framing and windowing. The magnitude of the FFT transform result is taken, squared, and then divided by the corresponding number of FFT points to obtain the power spectrum. The power spectrum is then converted to the Mel frequency scale through the Mel filter bank. The corresponding frequency is linear on the Mel frequency scale. The last step is to perform a logarithmic transformation on each element of the Mel spectrum to obtain the Mel spectrum.

[0040] Furthermore, the specific approach of S5 is:

[0041] Multi-scale channel attention module MS-CAM, the size of the input feature X is C×H×W, which is compressed into a C×1×1 tensor through global average pooling to extract the global statistical information of each channel; then, a point convolution is performed to compress the channel to obtain a dimension of The features are processed with batch normalization and ReLU activation function to increase nonlinearity. Then, the dimension of the features is restored back to C×1×1 through the second point convolution and then processed with batch normalization again.

[0042] The recovered features are fused with the input features, and the Sigmoid activation function is applied through the addition operation to generate the channel attention weight, whose size is C×H×W; finally, the input feature X is multiplied by the generated channel attention weight to obtain the output feature X′; the two branches on the right represent the channel attention of the global feature and the channel attention of the local feature, respectively. The channel attention calculation formula L(X) of the local feature is as follows:

[0043] L(X)=B(PWConv2(δ(B(PWConv1(X)))))

[0044] Among them, PWConv represents the point convolution operation, B represents batch normalization, and δ represents the ReLU activation function; this module can generate channel-specific attention weights by combining global statistical information and point convolution adjustment;

[0045] The multi-scale channel attention module MS-CAM is applied to the feature fusion of keystroke recognition. The specific approach is as follows:

[0046] First, by adjusting the multi-scale pooling and convolution structure, the characteristic differences of keystroke sounds in different frequency ranges are highlighted;

[0047] Secondly, the fusion of global timing features and local detail features in the keystroke signal was added to ensure that the convolutional neural network model can not only focus on the overall timing features of the keystroke segment, but also capture the local change detail features in the keystroke segment; an adaptive initial feature fusion strategy was introduced to make the initial fusion result more in line with the characteristics of the audio data; in addition, the iterative attention mechanism iAFF was adopted. iAFF uses the MS-CAM module for feature enhancement and fusion, and refines the feature fusion of the keystroke segment signal through multiple iterations.

[0048] Furthermore, the specific approach of S6 is:

[0049] A convolutional neural network (CNN) is used for deep learning model training. CNN consists of two convolutional layers and one fully connected layer, and each convolutional layer is followed by a ReLU activation function. At the same time, the size of the feature map is reduced through a pooling layer to reduce computational complexity and retain key features. During the deep learning model training process, the cross-entropy loss function is used to optimize the recognition results. After training, the deep learning model uses the Softmax function to generate recognition results in the output layer and selects the category with the highest probability as the final prediction result to complete the keystroke recognition task.

[0050] A computer terminal comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned keystroke recognition method based on acoustic eavesdropping is implemented.

[0051] A computer-readable medium stores a computer program thereon, wherein the computer program is executed by a processor to implement the above-mentioned keystroke recognition method based on acoustic eavesdropping.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] This paper extracts features from keystroke signals, effectively capturing their meaning. By implementing a feature fusion strategy and incorporating an attention mechanism, it assigns higher weights to certain features during feature fusion, achieving a deeper understanding of keystroke signals. Experimental results show that the model with feature fusion outperforms single features without fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0055] Figure 2 is a schematic diagram of a keystroke recognition method based on acoustic eavesdropping according to the present invention;

[0056] Figure 3 This is a Mel-map feature extraction flow chart of a keystroke recognition method based on acoustic eavesdropping according to the present invention;

[0057] Figure 4 Schematic diagram of the structure of a multi-scale channel attention module of a keystroke recognition method based on acoustic eavesdropping in the present invention;

[0058] Figure 5 Schematic diagram of the iterative attention feature fusion module structure of a keystroke recognition method based on acoustic eavesdropping of the present invention;

[0059] Figure 6 Schematic diagram of the performance accuracy and loss rate of a CNN model of a keystroke recognition method based on acoustic eavesdropping according to the present invention;

[0060] Figure 7 It is a frequency domain distribution diagram of a keystroke recognition method based on acoustic eavesdropping according to the present invention. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings and specific embodiments. Figure 1-2 As shown, a keystroke recognition method based on acoustic eavesdropping specifically includes the following steps:

[0062] S1. Obtain keystroke data: Obtain audio data from Harrison et al. in the paper "Harrison J, Toreini E, Mehrnezhad MA practical deep learning-based acoustic side channel attack on keyboards. 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 2023: 270-280." This data is collected in a controlled environment using highly sensitive equipment and reflects the audio features generated when the user taps the keyboard, providing a reliable data foundation for subsequent keystroke recognition.

[0063] S2, pre-processes the keystroke data obtained by S1;

[0064] S2.1, pre-emphasis, first, pre-emphasize the keystroke data obtained by S1. Pre-emphasis is an effective signal compensation technology whose main purpose is to enhance the high-frequency components of the signal. Its operation can be expressed as:

[0065] y[n]=x[n]-αx[n-1]

[0066] Where x[n] represents the original signal sequence, y[n] is the signal sequence after pre-emphasis processing, and the parameter α is the pre-emphasis coefficient;

[0067] S2.2, noise reduction, performs noise reduction on the result obtained by pre-emphasis in S2.1. First, the average value of the first five frames of keystroke data is taken to construct a noise model, and the spectrum is averaged to obtain the noise spectrum representing the background noise, that is:

[0068]

[0069] Where n is the number of frames used to calculate the average value, is the average value estimate of the noise spectrum, which is used to represent the noise intensity at frequency ω, and represents the frequency domain representation of the acoustic signal in the i-th frame;

[0070] In spectral subtraction, assuming that the noise interference in the original signal is additive, the collected keystroke data can be regarded as a linear superposition of the keystroke signal and the noise signal. Thus, by estimating the noise component and subtracting it from the keystroke data, a clean signal P that is closer to the original can be restored. N ,In order to further avoid the negative signal in spectrum subtraction, the denoised signal is

[0071]

[0072] Among them, P N (ω) represents the signal spectrum after noise reduction processing, that is, the estimated value of the clean signal. At frequency ω, it is the result of removing the noise signal from the keystroke data. Y(ω) represents the spectrum of the original keystroke data, which contains the mixed spectrum of the keystroke signal and the noise signal. N is the subtraction factor calculated based on the signal-to-noise ratio, represents the mean value estimate of the noise spectrum, Represents the final signal spectrum value after noise reduction, which is an estimate of the clean signal spectrum, β N It is used to determine the adjustment coefficient to avoid negative numbers when the amplitude of the signal after noise reduction is less than the noise amplitude. In order to adaptively adjust the noise estimation model, the noise spectrum is also updated according to the signal-to-noise ratio (SNR) of the calculated signal. The signal-to-noise ratio is calculated as follows:

[0073]

[0074] When the signal-to-noise ratio in a frame is lower than the set empirical threshold, the noise estimation spectrum is adaptively adjusted according to the noise level. The update calculation process of the noise estimation model is as follows:

[0075]

[0076] in, Represents the updated noise spectrum estimate, G N Represents a gain factor between 0 and 1, used to control the current noise estimate and the weight of the observed spectrum Y(ω) of the current frame in the noise estimation;

[0077] S3, keystroke event detection: Perform fast Fourier transform on the keystroke signal pre-processed in S2, convert the time domain into the frequency domain, and check the frequency domain of the keystroke signal. The calculation process is as follows:

[0078]

[0079] in, is the rotation factor, and its calculation formula is The keystroke signal is x(n), n=0,1,…,N-1;

[0080] Look at the unique characteristics of the keystroke signal frequency and calculate the spectral energy of the keystroke signal, using E k Indicates that the empirical threshold is used to determine whether to enter the keystroke process. When the E of any one of the two channels kWhen the signal exceeds the empirical threshold, it is considered that the keystroke process has begun, and this time point is used as the starting point of the keystroke segment. The signal of the following empirical threshold length is treated as a separate keystroke segment. After the key separation algorithm is applied, multiple keystroke segments are obtained, and each keystroke segment contains only one keystroke behavior.

[0081] S4, feature extraction: extract acoustic features from the keystroke fragments obtained in S3, including spectrogram and mel-spectrogram features;

[0082] Spectrogram feature extraction: Use the built-in spectrogram function in Matlab to calculate the short-time Fourier transform of the audio signal of the keystroke fragment to generate a spectrogram;

[0083] Mel-spectrogram feature extraction divides the keystroke segment into several frames, and the corresponding sample numbers of the frame length L and frame shift S are:

[0084] N=L×f s ,H=S×f s

[0085] Among them, f s is the signal sampling rate;

[0086] After the framing is completed, a Hanning window function is added to each frame, and then a fast Fourier transform is performed on each frame after framing and windowing. The magnitude of the FFT transform result is taken, squared, and then divided by the corresponding number of FFT points to obtain the power spectrum. The power spectrum is then converted to the Mel frequency scale through the Mel filter bank. The corresponding frequency is linear on the Mel frequency scale. The last step is to logarithmically transform each element of the Mel spectrum to obtain the Mel spectrum. The algorithm flow chart is as follows Figure 3 As shown;

[0087] S5, feature fusion and attention mechanism: Introducing a multi-scale channel attention module and an iterative attention feature fusion module to fuse spectrogram and mel-spectrogram features, and using the attention mechanism to strengthen the weight of key features such as the sound produced by fingers pressing the keyboard.

[0088] The structure diagram of the multi-scale channel attention module MS-CAM is as follows Figure 4 As shown, the size of the input feature X is C×H×W, which is compressed into a C×1×1 tensor after global average pooling to extract the global statistical information of each channel; then, a point convolution is performed to compress the channel to obtain a dimension of The features are processed with batch normalization and ReLU activation function to increase nonlinearity. Then, the dimension of the features is restored back to C×1×1 through the second point convolution and then processed with batch normalization again.

[0089] The recovered features are fused with the input features, and the Sigmoid activation function is applied through the addition operation to generate the channel attention weight, whose size is C×H×W; finally, the input feature X is multiplied by the generated channel attention weight to obtain the output feature X′; the two branches on the right represent the channel attention of the global feature and the channel attention of the local feature, respectively. The channel attention calculation formula L(X) of the local feature is as follows:

[0090] L(X)=B(PWConv2(δ(B(PWConv1(X)))))

[0091] Among them, PWConv represents the point convolution operation, B represents batch normalization, and δ represents the ReLU activation function; this module can generate attention weights for specific channels by combining global statistical information and point convolution adjustment, helping the model to better focus on important channel information and improve the expressiveness of features.

[0092] The multi-scale channel attention module MS-CAM is applied to the feature fusion of keystroke recognition. The specific approach is as follows:

[0093] First, by adjusting the multi-scale pooling and convolution structure, the characteristic differences of keystroke sounds in different frequency ranges are highlighted, so that the convolutional neural network model in S6 can more effectively capture different frequency and timing features;

[0094] Secondly, the fusion of global timing features and local detail features in the keystroke signal is added to ensure that the convolutional neural network model can not only focus on the overall timing features of the keystroke segment, but also capture the local change detail features in the keystroke segment; an adaptive initial feature fusion strategy is introduced to make the initial fusion result more consistent with the characteristics of the audio data, thereby providing higher quality input for the attention module; in addition, an iterative attention mechanism iAFF is adopted, such as Figure 5 As shown in the figure, iAFF uses the MS-CAM module twice for feature enhancement and fusion, and refines the feature fusion of the keystroke segment signal through multiple iterations, making it clearer and more comprehensive in expressing the temporal features and local features of the keystroke segment, thereby improving the model's robustness in dealing with dynamic noise changes and further enhancing the accuracy and stability of keystroke recognition.

[0095] S6, deep learning model training and recognition output: A convolutional neural network (CNN) is used for deep learning model training. CNN consists of two convolutional layers and one fully connected layer. Each convolutional layer is followed by a ReLU activation function to enhance the model's ability to express nonlinear features. At the same time, the size of the feature map is reduced through the pooling layer to reduce computational complexity and retain key features. During the deep learning model training process, the cross-entropy loss function is used to optimize the recognition results to improve the deep learning model's classification accuracy for keystroke data. After training, the deep learning model uses the Softmax function to generate recognition results in the output layer and selects the category with the highest probability as the final prediction result to complete the keystroke recognition task.

[0096] This paper extracts spectrogram and mel-map features from keystroke signals, effectively capturing both detailed and global features of the keystroke sound. Furthermore, by implementing a feature fusion strategy and incorporating an attention mechanism, it assigns higher weights to certain features during feature fusion, helping to improve the model's recognition accuracy and achieve a deeper understanding of keystroke behavior. This strategy not only enhances the model's ability to recognize subtle differences in keystroke signals but also improves generalization performance in complex environments.

[0097] Experiments were conducted using the audio data mentioned by Harrison et al. in the document "Harrison J, Toreini E, Mehrnezhad MA practical deep learning-based acoustic side channel attack on keyboards. 2023 IEEEEuropean Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 2023: 270-280." The experimental results show that the keyboard keystroke recognition method of the present invention achieved an accuracy of 87% and 89% respectively by training using spectrogram features and Mel-spectrogram features alone. This result verifies the effectiveness of single feature extraction technology in keyboard keystroke recognition. However, when the feature fusion strategy is adopted, especially when combined with the attention mechanism, the accuracy of the model is improved to 91.54%, as shown in Figure 2. Figure 6 As shown in the figure, it demonstrates the significant effect of feature fusion combined with attention mechanism in improving recognition accuracy.

[0098] In terms of data preprocessing, adaptive spectral subtraction is used for noise reduction, and the signal-to-noise ratio is set to 5. The noise model is adaptively adjusted according to the signal-to-noise ratio. Keystroke separation is performed based on spectral energy. Figure 7The spectrum diagram shows that the keystroke signal has unique frequency domain characteristics in the range of 1-1000 Hz, so the spectrum energy accumulation value in this range is calculated, and the empirical threshold is set to 15. When E k When it exceeds 15, it is considered that the keystroke signal has entered the contact wave peak, and the next 180ms is used as the keystroke segment to separate the keystroke from other invalid sounds, thereby obtaining the keystroke segment.

[0099] For feature extraction, the spectrogram feature extraction parameters are set to 512 for nfft, 18ms for frame length, 9ms for window shift, and a Hanning window function. Spectral features are calculated within the 0-8000 Hz range. For Mel-map feature extraction, the Mel filter is set to 40 for spectral features within the 0-3000 Hz range. The nfft, frame length, and window shift are the same as for the spectrogram feature. During feature fusion, the Mel-map is used as feature X, and the spectrogram is used as feature Y, which has a larger receptive field.

[0100] In terms of data set partitioning, we used 20 keystrokes out of 25 for model training and the remaining 5 keystrokes for validation and testing model performance. Data preprocessing and feature extraction were performed on Matlab R2023b, and model training was performed on PyCharm Community Edition 2020.2.1.

[0101] During the training process, the data size of each batch of the model is set to 32, the learning rate is set to 0.001, and the cross-entropy loss function is used to measure the difference between the true label and the predicted label, and the model parameters are updated accordingly in combination with the adam optimizer.

[0102] Although the above describes the illustrative specific embodiments of the present invention to facilitate understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A keystroke recognition method based on acoustic eavesdropping, characterized in that: The specific steps include: S1, obtaining keystroke data: using the mobile phone microphone to collect keystroke audio data and obtain the acoustic signal generated during the keystroke process; S2, keystroke data preprocessing: pre-emphasize the keystroke data obtained in S1 to enhance the high-frequency components of the signal, and use adaptive spectral subtraction to remove background noise to obtain a clean keystroke signal; S3, keystroke event detection: Perform fast Fourier transform on the keystroke signal preprocessed by S2, calculate the spectrum energy, and separate the keystroke signal into individual keystroke segments, ensuring that each keystroke segment contains only one keystroke behavior; S4, feature extraction: extract spectrogram and mel-spectrogram features for each keystroke segment, and extract acoustic features from the keystroke segment; S5, Feature Fusion and Attention Mechanism: Introducing a multi-scale channel attention module and an iterative attention feature fusion module to fuse spectrogram and mel-spectrogram features, and giving higher weights to key features through the attention mechanism; S6, inputs the fused feature data into the convolutional neural network model for training, uses the model to classify and identify the keystroke behavior, and finally obtains the keystroke recognition result.

2. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The keystroke data in S1 is collected using highly sensitive equipment in a controlled environment. The keystroke data contains the audio features generated when the user taps the keyboard.

3. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific method of preprocessing keystroke data in S2 is: S2.1 performs pre-emphasis processing on the keystroke data obtained by S1. The operation can be expressed as follows: y[n]=x[n]-αx[n-1] Where x[n] represents the original signal sequence, y[n] is the signal sequence after pre-emphasis processing, and the parameter α is the pre-emphasis coefficient; S2.2, the result obtained by pre-emphasis in S2.1 is subjected to noise reduction. First, the average value of the first five frames of keystroke data is taken to construct a noise model. The spectrum is averaged to obtain the noise spectrum representing the background noise, that is: Where n is the number of frames used to calculate the average value, is the average value estimate of the noise spectrum, which is used to represent the noise intensity at frequency ω, and represents the frequency domain representation of the acoustic signal in the i-th frame; In spectral subtraction, assuming that the noise interference in the original signal is additive, the collected keystroke data can be regarded as a linear superposition of the keystroke signal and the noise signal. Thus, by estimating the noise component and subtracting it from the keystroke data, a clean signal P that is closer to the original can be restored. N ,In order to further avoid the negative signal in spectrum subtraction, the denoised signal is Among them, P N (ω) represents the signal spectrum after noise reduction processing, that is, the estimated value of the clean signal. At frequency ω, it is the result of removing the noise signal from the keystroke data. Y(ω) represents the spectrum of the original keystroke data, which contains the mixed spectrum of the keystroke signal and the noise signal. N is the subtraction factor calculated based on the signal-to-noise ratio, represents the mean value estimate of the noise spectrum, Represents the final signal spectrum value after noise reduction, which is an estimate of the clean signal spectrum, β N It is used to determine the adjustment coefficient to avoid negative numbers when the amplitude of the signal after noise reduction is less than the noise amplitude. In order to adaptively adjust the noise estimation model, the noise spectrum is also updated according to the signal-to-noise ratio (SNR) of the calculated signal. The signal-to-noise ratio is calculated as follows: When the signal-to-noise ratio in a frame is lower than the set empirical threshold, the noise estimation spectrum is adaptively adjusted according to the noise level. The update calculation process of the noise estimation model is as follows: in, Represents the updated noise spectrum estimate, G N Represents a gain factor between 0 and 1, used to control the current noise estimate and the weight of the observed spectrum Y(ω) of the current frame in the noise estimation.

4. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S3 is: Perform a fast Fourier transform on the keystroke signal preprocessed by S2 to convert the time domain into the frequency domain. Check the frequency domain of the keystroke signal. The calculation process is as follows: in, is the rotation factor, and its calculation formula is The keystroke signal is x(n), n=0,1,…,N-1; Look at the unique characteristics of the keystroke signal frequency and calculate the spectral energy of the keystroke signal, using E k Indicates that the empirical threshold is used to determine whether to enter the keystroke process. When the E of any one of the two channels k When the empirical threshold is exceeded, it is considered that the keystroke process has begun, and this time point is used as the starting point of the keystroke segment. The signal of the following empirical threshold length is used as a separate keystroke segment. After the key separation algorithm, multiple keystroke segments are obtained, and each keystroke segment contains only one keystroke behavior.

5. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S4 is: Spectrogram feature extraction: Use the built-in spectrogram function in Matlab to calculate the short-time Fourier transform of the audio signal of the keystroke fragment to generate a spectrogram; Mel-spectrogram feature extraction divides the keystroke segment into several frames, and the corresponding sample numbers of the frame length L and frame shift S are: N=L×f s ,H=S×f s Among them, f s is the signal sampling rate; After the framing is completed, a Hanning window function is added to each frame, and then a fast Fourier transform is performed on each frame after framing and windowing. The magnitude of the FFT transform result is taken, squared, and then divided by the corresponding number of FFT points to obtain the power spectrum. The power spectrum is then converted to the Mel frequency scale through the Mel filter bank. The corresponding frequency is linear on the Mel frequency scale. The last step is to perform a logarithmic transformation on each element of the Mel spectrum to obtain the Mel spectrum.

6. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S5 is: Multi-scale channel attention module MS-CAM, the input feature X is of size C×H×W, which is compressed into a C×1×1 tensor through global average pooling to extract global statistical information of each channel; Next, the channel is compressed through a point convolution to obtain a dimension of The features are processed with batch normalization and ReLU activation function to increase nonlinearity. Then, the dimension of the features is restored back to C×1×1 through the second point convolution and then batch normalized again. The recovered features are fused with the input features, and the Sigmoid activation function is applied through the addition operation to generate the channel attention weight, whose size is C×H×W; finally, the input feature X is multiplied by the generated channel attention weight to obtain the output feature X′; The two branches on the right represent the channel attention of global features and the channel attention of local features respectively. The channel attention calculation formula L(X) of local features is as follows: L(X)=B(PWConv2(δ(B(PWConv1(X))))) Among them, PWConv represents the point convolution operation, B represents batch normalization, and δ represents the ReLU activation function; this module can generate channel-specific attention weights by combining global statistical information and point convolution adjustment; The multi-scale channel attention module MS-CAM is applied to the feature fusion of keystroke recognition. The specific approach is as follows: First, by adjusting the multi-scale pooling and convolution structure, the characteristic differences of keystroke sounds in different frequency ranges are highlighted; Secondly, the fusion of global timing features and local detail features in the keystroke signal was added to ensure that the convolutional neural network model can not only focus on the overall timing features of the keystroke segment, but also capture the local change detail features in the keystroke segment; an adaptive initial feature fusion strategy was introduced to make the initial fusion result more in line with the characteristics of the audio data; in addition, the iterative attention mechanism iAFF was adopted. iAFF uses the MS-CAM module for feature enhancement and fusion, and refines the feature fusion of the keystroke segment signal through multiple iterations.

7. The keystroke recognition method based on acoustic eavesdropping according to claim 1, characterized in that: The specific approach of S6 is: A convolutional neural network (CNN) is used for deep learning model training. CNN consists of two convolutional layers and one fully connected layer, with each convolutional layer followed by a ReLU activation function. Pooling layers are used to reduce the size of feature maps to reduce computational complexity while retaining key features. During deep learning model training, a cross-entropy loss function is used to optimize recognition results. After training is completed, the deep learning model uses the Softmax function to generate recognition results in the output layer and selects the category with the highest probability as the final prediction result to complete the keystroke recognition task.

8. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the keystroke recognition method based on acoustic eavesdropping as described in any one of claims 1 to 7 is implemented.

9. A computer-readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement a keystroke recognition method based on acoustic eavesdropping as described in any one of claims 1 to 7.

Images