Acoustic side channel attack detection and identification method

Abstract

The invention discloses an acoustic side channel attack detection and identification method, which comprises the following steps of: acquiring keystroke audio data when a user inputs a keyboard, and acquiring an acoustic signal generated in a keystroke process; the collected keystroke audio is detected and separated, and a target keystroke fragment is extracted; performing noise reduction processing on the keystroke fragment; extracting Mel time-frequency features from the denoised keystroke segments; self-supervised learning is adopted on the keystroke audio data without labels, spectrum masks are applied to keystroke features, a feature reconstruction task is designed, and a model is guided to obtain discriminative keystroke features in combination with discrimination task constraints; and carrying out model fine tuning by using a tagged keystroke sample, optimizing classification performance, and carrying out classification prediction and user input decoding on a keystroke fragment to obtain a final keystroke content restoration result. According to the method, the modeling capability of keystroke signals is remarkably improved, the generalization performance of downstream classification tasks is enhanced, the accuracy and robustness of keystroke classification are improved, and relatively high environmental robustness is shown.

Description

Technical Field

[0001] This invention relates to the field of side-channel attack technology, and in particular to an acoustic side-channel attack detection and identification method. Background Technology

[0002] Keyboard input, as one of the most important human-computer interaction methods in modern digital life, is widely used in key scenarios such as personal office work, enterprise business, financial operations, and government services, undertaking core functions such as password input, account verification, and sensitive information entry. With the widespread adoption of high-performance microphones in smartphones and other smart terminals, their ability to collect environmental acoustic information has continuously improved. While this technological advancement has driven the development of voice interaction, it has also brought new security risks. The acoustic signals generated by users during keystrokes may be captured and used to launch side-channel attacks, leading to the leakage of sensitive data. Existing research shows that attackers, relying solely on recording keystrokes through ordinary microphones, can reconstruct the keystrokes using the recordings, achieving an accuracy rate exceeding 90% in some scenarios, posing a serious threat to information security and user privacy. Especially in high-value scenarios such as offices, finance, and government affairs where physical keyboards are widely used, the privacy and system security risks posed by keyboard input leakage are becoming increasingly severe. Attackers do not need to directly access user devices; relying solely on passive recording, they can remotely reconstruct or infer user input, drawing significant attention from academia and industry to acoustic side-channel attacks.

[0003] Some work focuses on analyzing behavioral information during keystrokes. They utilize the temporal characteristics between keystroke events (such as key delays, press-and-release time differences, etc.) to construct time series models to reconstruct sensitive inputs, for example, by identifying consecutive keystroke events through cross-correlation analysis.

[0004] However, such methods have limited generalization ability in scenarios with varying input patterns or diverse user behaviors. Meanwhile, another type of research attempts to reconstruct the geometric location of keystrokes by analyzing the differences in keystroke propagation across multiple microphones (such as time of arrival, sound intensity attenuation, and phase difference). The advantage of this type of method is its insensitivity to speech content and acoustic features, and it is theoretically applicable to all key inputs. It is worth noting that in actual deployment, even slight keyboard movements or environmental changes can cause drift in key parameters such as time of arrival, thus affecting positioning accuracy and the overall robustness of the model.

[0005] In contrast, frequency domain modeling based on deep learning has demonstrated stronger feature extraction capabilities in recent years. For example, by converting keystroke audio into Mel spectrograms and incorporating attention mechanisms, high recognition accuracy can be achieved under specific experimental conditions. However, to obtain sufficient generalization ability, model training typically requires large-scale labeled data, posing a significant challenge to practical deployment.

[0006] Therefore, in order to solve the problems of difficulty in detecting keystroke fragments under noise interference, insufficient feature modeling capabilities, and strong dependence on large-scale labeled data in existing technologies, it is an urgent technical problem for those skilled in the art to provide an acoustic side-channel attack detection and identification method that can ensure model recognition accuracy and improve its adaptability and deployment flexibility in complex real input environments. Summary of the Invention

[0007] In view of this, the present invention provides a method for detecting and identifying acoustic side-channel attacks.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] An acoustic side-channel attack detection and identification method includes the following steps:

[0010] S1, Keystroke Audio Collection:

[0011] Use a mobile phone or other terminal with a high-precision microphone to collect the keystroke audio data when the user inputs on the keyboard and obtain the acoustic signals generated during the keystroke process;

[0012] S2, Keystroke Event Detection and Separation:

[0013] Based on a multi-scale energy fusion strategy, the keystroke audio collected in step S1 is detected and separated to extract the target keystroke segment from the complex background sound, ensuring that each segment contains only one keystroke.

[0014] S3, Keystroke Audio Denoising:

[0015] The keystroke fragments obtained in step S2 are subjected to noise reduction processing to remove environmental noise and improve the quality of the target keystroke signal;

[0016] S4, Keystroke Feature Extraction:

[0017] Mel time-frequency features are extracted from the denoised keystroke segments to obtain a high-dimensional acoustic feature representation;

[0018] S5, Self-supervised Pre-training:

[0019] Self-supervised learning is employed on unlabeled keystroke audio data. A spectral mask is applied to the keystroke features and a feature reconstruction task is designed. The model is used to recover the masked time-frequency regions, thereby learning the key feature representations in the keystroke audio. At the same time, the model is guided to obtain discriminative keystroke features by combining discrimination task constraints.

[0020] S6, fine-tuning and keystroke recognition:

[0021] Based on the pre-trained model obtained in step S5, the model is fine-tuned using labeled keystroke samples to optimize classification performance. The keystroke fragments are then classified, predicted, and decoded to obtain the final keystroke content reconstruction result.

[0022] Preferably, in step S1, the keystroke data is collected by a common commercial smartphone with a built-in high-sensitivity microphone that supports mono or multi-channel audio storage.

[0023] The keystroke data includes sound leaks generated by the user during keyboard input.

[0024] Preferably, step S2, the detection and separation of keystroke events, specifically includes:

[0025] S2.1, Perform continuous wavelet transform on the input keystroke audio signal to obtain time-frequency coefficients at different scales;

[0026] S2.2 Calculate the wavelet energy spectrum and assign different weights to specific frequency ranges to highlight the high-frequency energy region unique to keystrokes;

[0027] S2.3 Calculate the average noise energy based on the previous few frames of audio and set a threshold as the trigger condition for keystroke events; when the energy difference value exceeds the threshold, it is determined that a keystroke event may occur.

[0028] S2.4 After detecting a keystroke candidate point, search for the maximum energy value within a certain range before and after it as the keystroke peak value, and introduce an offset parameter to ensure that the captured keystroke segment completely covers the contact peak, the pressing peak and the release peak.

[0029] S2.5 To avoid short-duration noise pulses being misjudged as keystroke events, a minimum keystroke time interval is set; only when the interval between adjacent candidate keystrokes is greater than this threshold is it determined to be a valid keystroke, and the corresponding audio segment is separated and saved.

[0030] Preferably, step S3 specifically includes:

[0031] The keystroke segments obtained in step S2 are denoised. During the denoising process, the signal is divided into a series of 20ms frames for processing. Several silent frames before the keystroke event are selected and treated as noise samples. Then, the spectra of these frames are averaged to obtain the noise spectrum representing the background noise. The calculation formula is as follows:

[0032] ,

[0033] in, This represents the estimated noise spectrum, while The representative of the first Frequency domain representation of intra-frame acoustic signals;

[0034] Among them, the clean signal is closer to the original. The calculation formula is as follows:

[0035] ,

[0036] in, It is a subtraction factor calculated based on the signal-to-noise ratio. It is an adjustment factor used to determine whether to avoid negative values ​​when the amplitude of the noise-reduced signal is less than the amplitude of the noise.

[0037] When a frame is detected that does not belong to a keystroke segment, the noise estimation model is updated according to the formula, and the noise estimation spectrum is adaptively adjusted based on the noise level:

[0038] ,

[0039] in, It is a smoothing factor used to control the update rate of noise estimation.

[0040] Preferably, in step S4, the keystroke feature extraction specifically includes:

[0041] S4.1, Pre-weighting treatment:

[0042] Pre-emphasis is applied to keystroke segments to boost high-frequency components and weaken low-frequency components. The calculation formula is as follows:

[0043]

[0044] in, The original signal, The signal after pre-emphasis. This is the pre-emphasis coefficient;

[0045] S4.2, framing and windowing:

[0046] The preprocessed signal is divided into several frames, with the frame length and frame shift set according to the sampling rate. Each frame signal is multiplied by a window function.

[0047] S4.3, Short-Time Fourier Transform (STFT):

[0048] Perform STFT on each frame of the signal to obtain its spectral representation:

[0049] ,

[0050] in, For frame length, For frame shift, Indexed by frequency bin;

[0051] S4.4, Power Spectrum Calculation and Mel Filtering:

[0052] Calculate the power spectrum for each frame The energy distribution on the Mel frequency scale is obtained by weighting the energy using a triangular Mel filter bank; The output of each Mel filter is:

[0053] ,

[0054] in, For the first The frequency response function of a Mel filter.

[0055] S4.5, Mel frequency mapping:

[0056] The linear frequency is calculated using the following formula. Mapped to Mel frequency

[0057] ;

[0059] S4.6, Feature Concatenation and Representation:

[0060] The logarithms of the energy outputs of all filters are taken and concatenated to form a Mel spectrum, which yields a high-dimensional acoustic feature representation of the keystroke fragments and serves as input for subsequent models.

[0061] Preferably, in step S5, the self-supervised pre-training specifically includes:

[0062] S5.1, Model Building:

[0063] A network structure based on Masked Auto Encoder is constructed. The input is a Mel spectrogram divided into frames, and a masking strategy is adopted to encode only the unmasked patches. Two parallel linear prediction heads are set at the decoder output to perform reconstruction and discrimination tasks respectively.

[0064] S5.2, Reconstruction Task:

[0065] At the decoder end, a reconstruction prediction head is set up, and the recovery error of the mask patch is calculated using the mean square error loss:

[0066] ,

[0067] in, For the first The reconstruction result of each mask patch. This is the original data, where M is the number of mask patches;

[0068] S5.3, Judgment Task:

[0069] By setting up a discriminative prediction head in parallel and introducing a contrastive learning loss, the model is constrained to aggregate positive sample features of the same keystroke and separate negative sample features of different keystrokes in the latent space, thereby learning a discriminative representation.

[0070] S5.4, Joint Optimization:

[0071] The total pre-training loss is the weighted sum of the reconstruction loss and the discrimination loss:

[0072] .

[0074] Preferably, in step S5.3, the loss function is in the form of:

[0075] ,

[0076] in, For the classification head prediction vector, It is the first An original patch is embedded.

[0077] Preferably, in step S6, the fine-tuning and keystroke content restoration specifically includes:

[0078] S6.1, Encoder Migration:

[0079] During the fine-tuning phase, the pre-trained encoder structure and its weights are retained and used as a feature extractor while the decoder is discarded, thus preserving the deep feature representation capabilities learned by the encoder during pre-training.

[0080] S6.2, Classification layer construction:

[0081] A global pooling layer is added at the end of the encoder, and a fully connected classification head is connected to it. Finally, a Softmax layer is connected to aggregate the temporal features of keystrokes into a global vector representation, thereby realizing the classification of keystroke fragments and the decoding of input content.

[0082] S6.3, Training and Optimization:

[0083] A smaller learning rate is used in the fine-tuning phase, the cross-entropy loss function is used for classification supervision, and an early stopping mechanism is introduced during training.

[0084] A computer terminal, the terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0085] A computer-readable medium having a computer program stored thereon.

[0086] The present invention achieves the following technical effects compared to the prior art:

[0087] (1) This invention introduces a self-supervised pre-training framework, which combines a spectrum masking mechanism with a feature reconstruction task, enabling the model to automatically learn key feature representations in keystroke audio and significantly improve the modeling ability of keystroke signals under unlabeled data conditions.

[0088] (2) By setting up reconstruction and discrimination tasks in parallel, the model can not only recover the occluded time-frequency region, but also learn discriminative keystroke features, thereby enhancing the generalization performance of downstream classification tasks.

[0089] (3) In the fine-tuning stage, the present invention further optimizes the model parameters by using labeled keystroke audio samples, thereby improving the accuracy and robustness of keystroke classification;

[0090] (4) The present invention constructs a complete keyboard acoustic side-channel attack process and conducts experiments in various mainstream keyboards and real environments. The results show that it is superior to existing baseline methods in terms of core indicators such as accuracy, F1-score and recall.

[0091] (5) The present invention exhibits good stability and adaptability under various noise levels, different recording distances and various types of microphones, demonstrating strong environmental robustness. Attached Figure Description

[0092] Figure 1 This is a schematic diagram illustrating the steps of the method for restoring keystroke content;

[0093] Figure 2 This is a flowchart of the method for restoring keystroke content;

[0094] Figure 3 This is a diagram showing the comparison of keystroke audio waveforms before and after noise reduction;

[0095] in, Figure 3 (a) Before noise reduction; Figure 3 (b) shows the result after noise reduction;

[0096] Figure 4 This is a schematic diagram illustrating the changes in reconstruction evaluation metrics during pre-training;

[0097] Figure 5 This is a diagram illustrating the changes in accuracy between the top 1 and top 5 classifications during pre-training.

[0098] Figure 6 This is a visual comparison of the reconstruction of the Meltu mask;

[0099] in, Figure 6 (a) is the original Merr diagram; Figure 6(b) is Meltu after being concealed; Figure 6 (c) To reconstruct Melto;

[0100] Figure 7 This is a performance graph of the method provided by the present invention on different keyboard data types;

[0101] Figure 8 This is a performance comparison chart with and without pre-training provided in Embodiment 1 of the present invention;

[0102] Figure 9 This is a performance comparison chart at different distances provided in Embodiment 2 of the present invention;

[0103] Figure 10 This is a performance comparison chart under different noise levels provided in Embodiment 3 of the present invention;

[0104] Figure 11 This is a performance comparison chart for different mobile phone types provided in Embodiment 4 of the present invention. Detailed Implementation

[0105] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0106] like Figure 1-2 As shown, this invention discloses a method for detecting and identifying acoustic side-channel attacks, which specifically includes the following steps:

[0107] S1, Get keystroke data

[0108] To support self-supervised pre-training and keystroke audio classification tasks, this invention constructs two datasets: a pre-training dataset for self-supervised learning and a classification dataset for fine-tuning. Recording is performed using the built-in microphone of a smartphone, placed near the keyboard. To eliminate the effects of desktop vibrations, a folded piece of fiber cloth is placed under the phone, ensuring that the collected sound focuses more on the airborne portion.

[0109] S1.1, Construction of the pre-trained dataset

[0110] In a real office environment, this invention continuously recorded several hours of keystroke audio for different types of keyboards (including mechanical, laptop, and membrane keyboards) and multiple users; no guidance was provided to the users during the recording process.

[0111] S1.2, Construction of the classification dataset

[0112] To evaluate the proposed method's ability to perform keystroke inference attacks in real-world scenarios, a keyboard keystroke classification dataset was constructed for fine-tuning. This dataset includes Mechanical 1 (AULA F99), Mechanical 2 (Nuphy Air75), Laptop1 (MacBook Pro 14-inch 2024), Laptop2 (R9000P 2023), Membrane1 (MX Keys Mini), and Membrane2 (MK275). Data collection was conducted in an office environment with approximately 26dB of background white noise. Each key was pressed 25 times consecutively, with varying pressure and finger placement. A single audio file recorded all keystroke samples for that key.

[0113] In addition, keystroke audio data from Harrison et al. at "https: / / github.com / JBFH-Dev / Keystroke-Datasets." was introduced, denoted as the HRS dataset.

[0114] S2, detects and isolates keystroke events from keystroke audio;

[0115] S2.1, Continuous wavelet transform analysis,

[0116] A continuous wavelet transform is performed on the input keystroke audio signal to obtain time-frequency coefficients at different scales. The selected mother wavelet function is either Mexican Hat or Complex Morlet, used to capture high-frequency abrupt changes or mid-to-low-frequency sustained features.

[0117] S2.2, Energy Weighting and Differential Processing

[0118] Calculate the wavelet energy spectrum and assign different weights to specific frequency ranges where the keystroke audio is located in order to highlight the high-frequency energy region unique to keystrokes;

[0119] Further differential operations are performed on the energy curve to enhance the abrupt change characteristics of keystroke events.

[0120] S2.3 Noise Threshold Setting and Detection

[0121] The average noise energy is calculated based on the preceding frames of the keystroke audio, and empirical and dynamic threshold changes are set as triggering conditions for keystroke events. When the energy difference exceeds the threshold, it is determined that a keystroke event may have occurred.

[0122] S2.4, Keystroke Peak Positioning and Offset Correction

[0123] After detecting a candidate keystroke point, the maximum energy value within a certain range before and after it is searched as the keystroke peak value, and an offset parameter is introduced to ensure that the captured keystroke segment completely covers the contact peak, the press peak, and the release peak.

[0124] S2.5, Minimum Time Interval Constraint and Fragment Separation

[0125] To avoid short-duration noise pulses being misidentified as keystroke events, a minimum keystroke time interval is set. Only when the interval between adjacent candidate keystrokes exceeds this threshold is it considered a valid keystroke, and the corresponding audio segment is separated and saved. A total of 5618 valid keystroke events were separated from the recordings in the pre-training data using a keystroke event detection and separation algorithm.

[0126] S3, Denoise the keystroke fragments acquired in S2;

[0127] During the denoising process, the signal is divided into a series of 20ms frames for processing. The first 5 silent frames before the keystroke event are selected and treated as noise samples.

[0128] The spectra of these frames are then averaged to obtain the noise spectrum representing the background noise, calculated using the following formula:

[0129] ,

[0130] in, This represents the estimated noise spectrum, while The representative of the first Frequency domain representation of intra-frame acoustic signals.

[0131] In spectral subtraction, it is assumed that the noise interference in the original signal is additive, and the collected acoustic signal can be viewed as a linear superposition of the keystroke signal and the noise signal. Furthermore, the noise component is estimated and subtracted from the signal to recover a cleaner signal that more closely approximates the original. The calculation formula is as follows:

[0132] ,

[0133] in, It is a subtraction factor calculated based on the signal-to-noise ratio. It is an adjustment factor used to determine whether to avoid negative values ​​when the amplitude of the noise-reduced signal is less than the amplitude of the noise.

[0134] When a frame is detected that does not belong to a keystroke segment, we update the noise estimation model according to the following formula, and adaptively adjust the noise estimation spectrum based on the noise level:

[0135] ,

[0136] in, This is a smoothing factor used to control the update rate of noise estimation. After the original recording file is processed by the noise reduction algorithm, the keystroke waveforms that were originally masked by noise become clear, and the time-domain images without keystrokes are close to silent, such as... Figure 3 As shown.

[0137] S4, extract the acoustic features of the keystroke segments from the denoised keystroke segments;

[0138] S4.1, Pre-weighting treatment

[0139] The keystroke segment signal is pre-emphasized to enhance high-frequency components and weaken low-frequency components.

[0140] The calculation formula is as follows:

[0141]

[0142] in, The original signal, The signal after pre-emphasis. The pre-emphasis coefficient is set to 0.97.

[0143] S4.2, framing and windowing

[0144] The preprocessed keystroke segment signal was divided into several frames. Based on the trade-off between "information completeness" and "minimizing noise interference", we chose 16ms as the standard window length with 50% window overlap. A Hanning window was applied to each frame to reduce spectral leakage caused by inter-frame cutting.

[0145] S4.3, Short-Time Fourier Transform

[0146] Perform STFT on each frame of the signal to obtain its spectral representation:

[0147] ,

[0148] in, For frame length, For frame shift, For frequency bin index.

[0149] S4.4, Power Spectrum Calculation and Mel Filtering: Calculate the power spectrum for each frame. The energy distribution on the Mel frequency scale is obtained by weighting the energy distribution using a triangular Mel filter bank.

[0150] No. The output of each Mel filter is:

[0151] ,

[0152] in, For the first Frequency response function of a Mel filter.

[0153] S4.5, Mel frequency mapping, uses the following formula to map linear frequencies. Mapped to Mel frequency:

[0154]

[0155] S4.6, Feature Concatenation and Representation

[0156] The logarithm of the energy output from all filters is taken and concatenated to form a Mel spectrum, thereby obtaining a high-dimensional acoustic feature representation of the keystroke fragment, which serves as the input for subsequent training and classification models.

[0157] S5, self-supervised pre-training of the model;

[0158] S5.1, Model Building

[0159] A network structure based on MAE is constructed. The input is a Mel spectrogram divided into frames. A masking strategy along the time dimension is adopted, and only the unmasked patches are encoded. The decoder is used to reconstruct the spectrogram during the pre-training stage. A 6-layer encoder and a 2-layer decoder are used, with both the encoder and decoder using 12 heads and a width of 768.

[0160] During training, the model receives the occluded Mel spectrogram as input and performs two tasks in sequence: local reconstruction and contrast discrimination.

[0161] S5.2, Reconstruction Task

[0162] At the decoder end, a reconstruction prediction head is set up, and the recovery error of the mask patch is calculated using MSE:

[0163] ,

[0164] in, For the first The reconstruction result of each mask patch. This represents the original data, and M is the number of mask patches. This task aims to recover the occluded time-frequency regions and enhance the model's contextual modeling capabilities.

[0165] S5.3, Discrimination Task

[0166] The discriminative prediction head is set up in parallel, and the contrastive learning loss InfoNCE is introduced to constrain the model to aggregate positive sample features of the same keystroke and separate negative sample features of different keystrokes in the latent space, thereby learning a discriminative representation.

[0167] The loss function is in the form of:

[0168] ,

[0169] in, For the classification head prediction vector, It is the first The original patch embedding is used, and the loss function aims to enhance the model's discriminative ability to identify keystroke signals during reconstruction, thereby improving the representation quality of the target fragment.

[0170] S5.4, Joint Optimization

[0171] The total pre-training loss is the weighted sum of the reconstruction loss and the discrimination loss:

[0172] ,

[0173] in, To highlight the dominant role of the reconstruction task, the Adam optimizer was used during training with a learning rate of 1e-4, a batch size of 32, and 1000 iterations. After each epoch, the model's performance was evaluated on a fixed set of validation samples. In the reconstruction task, MSE and MAE were used to measure the error between the predicted result and the original value. In the discrimination task, Top-1 and Top-5 accuracy were used to measure the accuracy of the discrimination. The performance curves for the two tasks are shown below. Figure 4 and Figure 5 As shown, the model consistently improves in both reconstruction and discrimination capabilities, demonstrating stable performance. We use the best model from the pre-training process for reconstruction visualization, as shown below. Figure 6 As shown, the model successfully recovered the spectral characteristics of the masked region, and even in high-frequency and low-energy regions, the model was able to restore details well.

[0174] S6, model fine-tuning and keystroke recognition;

[0175] S6.1, Encoder Migration

[0176] The pre-trained encoder structure and weights are retained as feature extractors, while the decoder is discarded, thus preserving the deep feature representation capabilities learned by the encoder during pre-training.

[0177] S6.2, Classification Construction

[0178] A global pooling layer is added at the end of the encoder, connected to a fully connected classification head, and finally connected to a Softmax layer to aggregate the temporal features of keystrokes into a global vector representation, thereby enabling keystroke category prediction and input decoding.

[0179] S6.3, Training and Optimization

[0180] A smaller learning rate is used during the fine-tuning phase to avoid drastic perturbations to the pre-trained weights. Classification supervision employs the cross-entropy loss function (the dataset is randomly split 5 times and tested), and an early stopping mechanism is introduced during training with a patience value of 15 to prevent overfitting.

[0181] The evaluation results on classification datasets are shown in Figure 7. The proposed model achieves a classification accuracy of over 92% on all datasets. Among them, it performs best on mechanical keyboard keystroke audio, thanks to the stable multi-band resonance characteristics brought by its independent axis structure, achieving the highest accuracy of over 99%. Although laptop keyboard audio is mainly concentrated in the mid-low frequency range, the model still achieves a high recognition rate due to the advantages of its manufacturing process.

[0182] In contrast, membrane keyboards, due to the rubber dome structure limiting high-frequency components, exhibit greater signal fluctuations and relatively weaker classification performance, with an average minimum accuracy of 92.16%.

[0183] In particular, the model achieved a recognition accuracy of over 98% on the public dataset HRS, further validating the generalization and robustness of the invention.

[0184] Example 1: The impact of self-supervised pre-training on the accuracy of keystroke classification tasks

[0185] To verify the effectiveness of the self-supervised pre-training strategy for the keystroke recognition model, this embodiment designed a comparative experiment to compare the performance differences of the model under conditions of introducing pre-training and not introducing pre-training.

[0186] Experimental conditions: The experiment was conducted on the aforementioned classification dataset, which covers various keyboard types, including mechanical keyboards, laptop keyboards, and membrane keyboards. The experimental environment, keyboard configuration, and recording equipment were kept consistent to ensure the reliability of the comparison results.

[0187] Experimental Procedure: Under the same dataset and experimental conditions, two types of models were trained: one was a model without self-supervised pre-training, and the other was a model that underwent self-supervised pre-training on unlabeled keystroke audio data and was further fine-tuned. Both models were tested on the same classification task, and the classification accuracy for each keyboard type was recorded and compared.

[0188] Results and Analysis:

[0189] Experimental results are as follows Figure 8As shown in the figure, overall, the model with self-supervised pre-training significantly outperformed the untrained model in classification accuracy across all keyboard types. On mechanical keyboard data, the pre-trained model achieved a near 99% accuracy, demonstrating the best performance. On laptop keyboard data, despite the predominantly low-to-mid-frequency keystrokes, the pre-trained model maintained a stable accuracy advantage. On membrane keyboard data, the baseline model performed poorly due to the rubber dome structure limiting the high-frequency components of the keystrokes, while the pre-trained model showed the greatest improvement, with an accuracy gain of up to 11.52%. Furthermore, on the public dataset HRS, the pre-trained model achieved a classification accuracy exceeding 98%, further demonstrating the method's generalization ability and robustness across different datasets.

[0190] The experimental results show that self-supervised pre-training can significantly enhance the model's ability to model keystroke audio features, effectively improve the accuracy and adaptability of classification tasks, and especially demonstrate stronger robustness in the context of thin-film keyboards with unstable signal features.

[0191] Example 2: The impact of different distances on the accuracy of keystroke classification tasks

[0192] To verify the impact of the distance between the recording device and the keyboard on keystroke classification performance, this embodiment designed a set of comparative experiments.

[0193] Experimental Conditions: To verify the impact of the distance between the recording device and the keyboard on keystroke classification performance, this embodiment designed a set of comparative experiments. In the experiment, two identical commercial smartphones (Xiaomi 15) were used as the recording devices, both with built-in high-sensitivity microphones; a mechanical keyboard was used; the experimental setting was a typical office environment, maintaining a stable background noise level. The two phones were placed at different specified distances and recorded simultaneously to ensure that comparative audio signals were obtained under the same keystroke conditions. Specifically, six distance combinations were set: (10cm, 20cm), (10cm, 50cm), (10cm, 100cm), (20cm, 50cm), (20cm, 100cm), and (50cm, 100cm).

[0194] Experimental Procedure: In each experiment, subjects performed preset keystrokes on a mechanical keyboard 1 according to a specified input task. Two smartphones recorded the acoustic signals generated during the keystrokes. The recorded audio data underwent preprocessing, keystroke event detection and separation, feature extraction, and classification model recognition to obtain the corresponding classification accuracy results.

[0195] Results and Analysis: The experimental results are as follows... Figure 9As shown (without pre-training), when the recording device is close to the keyboard (10cm, 20cm), the keystroke classification accuracy is generally high, with the best performance at 20cm. At 10cm, the accuracy is slightly lower, likely due to interference from desktop vibrations during close-range recording. As the distance increases further to 50cm and 100cm, the keystroke sound signal gradually attenuates, and the classification performance decreases accordingly, with accuracy lower than under close-range conditions. Comprehensive analysis indicates that in practical applications, setting the distance between the recording device and the keyboard to approximately 20cm can effectively reduce vibration interference while maintaining recording quality, thus achieving a more ideal keystroke classification accuracy and stability.

[0196] Example 3: The impact of different ambient noise levels on keystroke recognition performance

[0197] To systematically evaluate the impact of environmental noise on the performance of the keystroke recognition model, this embodiment conducted experiments under various noise levels and types. By comparing noise conditions in different scenarios, the robustness and adaptability of the model in complex environments were examined.

[0198] Experimental conditions: A mechanical keyboard was used as the input device, and a commercial smartphone (Xiaomi 15) with a high-sensitivity microphone was used for recording. Three typical environmental scenarios were set up: apartment, laboratory, and conference room. The apartment scenario included white noise (25 dB); the laboratory scenario included white noise (22 dB), other keystroke background noise (45 dB), and machine operation noise (43 dB); and the conference room scenario included white noise (31 dB), human conversation (60 dB), and machine operation noise (42 dB). The noise intensity ranged from 22 dB to 60 dB.

[0199] Experimental Procedure: Keystroke audio data was collected under different noise conditions. The collected raw audio was first preprocessed and keystroke segments were separated. Then, Mel time-frequency features were extracted, and finally, the data were input into a classification model for recognition and evaluation. By comparing the classification results under different noise scenarios and intensities, the impact of environmental noise on keystroke recognition performance was analyzed.

[0200] Results and Analysis: The experimental results are as follows... Figure 10As shown, in the three scenarios, the accuracy and recall of keystroke classification decreased with increasing noise intensity, but the overall accuracy remained at a high level. Denoising significantly improved model performance, especially under white noise and machine noise conditions. The impact of different noise types varied; strong human voice dialogue noise had the greatest impact on keystroke recognition performance, while low-intensity white noise had relatively less impact. Further analysis showed that even in certain noisy environments, the model could still maintain a certain level of keystroke recognition capability, indicating that the method proposed in this invention has strong robustness and environmental adaptability, and can play an effective role in real-world applications.

[0201] Example 4: The impact of different recording devices on keystroke recognition performance

[0202] To evaluate the robustness of the keystroke recognition model proposed in this invention under different sound acquisition device conditions, this embodiment conducted a comparative experiment on the microphone acquisition performance of multiple smartphones, aiming to verify the stability of the model under hardware differences.

[0203] Experimental conditions: Five mainstream smartphones were selected as recording devices in the experiment: Xiaomi 15, Vivo S15, Reno 14, Redmi 13 Pro+, and Nova 12. Mechanical keyboard 1 was used. The experimental environment was kept consistent with the data collection method of the classification dataset to ensure the comparability of the test results.

[0204] Experimental Procedure: Under the same experimental environment, subjects used a mechanical keyboard 1 to perform a standardized keystroke input task. Different mobile phone models were used to record keystroke audio data. The recorded data then underwent uniform preprocessing, keystroke event detection and separation, and feature extraction. Finally, the data was input into a classification model for recognition. For each recording device, the accuracy, F1-score, and recall of keystroke recognition were calculated to evaluate the model's performance under different data acquisition conditions.

[0205] Results and Analysis: The experimental results are as follows... Figure 11 As shown, without pre-training, the model performed consistently on keystroke data recorded on five different mobile phones, maintaining a classification accuracy of around 95%, with stable F1-score and recall. These results demonstrate that the keystroke recognition model of this invention exhibits good robustness and consistency across different data acquisition devices, and its overall recognition performance is not significantly affected by hardware differences.

[0206] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. An acoustic side-channel attack detection identification method, characterized in that, Comprise the following steps: S1, keystroke audio collection: Collect the keystroke audio data of the user when inputting the keyboard by using the mobile phone or other terminal with high-precision microphone, and obtain the acoustic signal generated in the keystroke process; S2, keystroke event detection and separation: Based on the multi-scale energy fusion strategy, the keystroke audio collected in the step S1 is detected and separated, the target keystroke segment is extracted from the complex background sound, and it is ensured that each segment only contains one keystroke behavior; S3, keystroke audio denoising: The keystroke segment obtained in the step S2 is denoised to remove the environmental noise and improve the quality of the target keystroke signal; S4, keystroke feature extraction: The mel-frequency cepstral coefficient of the denoised keystroke segment is extracted to obtain a high-dimensional acoustic feature representation; S5, self-supervised pre-training: On the unlabeled keystroke audio data, self-supervised learning is adopted to apply a spectral mask to the keystroke feature and design a feature reconstruction task, and the model is used to recover the masked time-frequency area, so as to learn the key feature representation in the keystroke audio, and at the same time, the discriminative keystroke feature is obtained by combining the discriminative task constraint; S6, fine-tuning and keystroke content recognition: On the basis of the pre-trained model obtained in the step S5, the model is fine-tuned by using the labeled keystroke sample to optimize the classification performance, the keystroke segment is classified and predicted, and the user input is decoded to obtain the final keystroke content restoration result.

2. The method of claim 1, wherein, In the step S1, the keystroke data is collected by a common commercial intelligent mobile phone, the intelligent mobile phone is built-in high-sensitivity microphone, and supports single-channel or multi-channel audio storage; The keystroke data contains the sound leakage generated by the user in the keyboard input process.

3. The method of claim 1, wherein, In the step S2, the keystroke event detection and separation specifically comprises: S2.1, the input keystroke audio signal is subjected to continuous wavelet transform to obtain time-frequency coefficients at different scales; S2.2, the wavelet energy spectrum is calculated, and different weights are given to the specific frequency range to highlight the high-frequency energy area of the keystroke; S2.3, the average noise energy is calculated according to the front several frames of audio, and a threshold is set as the trigger condition of the keystroke event; when the energy difference value exceeds the threshold, it is determined that the keystroke event may occur; S2.4, after detecting the keystroke candidate point, the maximum energy value within a certain range before and after the keystroke candidate point is searched as the keystroke peak value, and an offset parameter is introduced to ensure that the intercepted keystroke segment completely covers the contact peak, the pressing peak and the release peak; S2.5, in order to avoid short noise pulses being misjudged as keystroke events, a minimum keystroke time interval is set; only when the interval between adjacent candidate keystrokes is greater than the threshold, the keystroke is determined to be effective, and the corresponding audio segment is separated and saved.

4. The method of claim 1, wherein, The step S3 specifically comprises: The keystroke segment obtained in the step S2 is denoised, and in the denoising process, the signal is divided into a series of 20ms frames for processing, a plurality of silent frames before the keystroke event occur are selected, the silent segment is regarded as a noise sample, then the spectrum of these frames is averaged to obtain a noise spectrum representing the background noise, and the calculation formula is as follows: ， wherein represents an estimated noise spectrum, while represents the first a frequency domain representation of the intra acoustic signal; Where the closer to the original clean signal The calculation formula is as follows: ， wherein, is a subtraction factor calculated from the signal-to-noise ratio, is an adjustment coefficient for determining when to avoid negative numbers when the amplitude of the signal after noise reduction is less than the amplitude of the noise. When a frame is scanned that does not belong to the keystroke segment, the noise estimation model is updated according to the formula, and the noise estimation spectrum is adaptively adjusted according to the noise size: ， wherein is a smoothing factor for controlling the update speed of the noise estimate.

5. The method of claim 1, wherein, The step S4 specifically includes keystroke feature extraction: S4.1, pre-emphasis processing: The keystroke segment is pre-emphasized to enhance the high-frequency component and weaken the low-frequency component, and the calculation formula is: wherein is the original signal, is the pre-emphasized signal, is the pre-emphasis coefficient; S4.2, frame division and windowing: The preprocessed signal is divided into several frames, and the frame length and frame shift are set according to the sampling rate. Each frame of signal is multiplied by a window function; S4.3, short-time Fourier transform STFT: STFT is performed on each frame of signal to obtain its spectral representation: ， wherein, is the frame length, is the frame shift, is the frequency bin index; S4.4, power spectrum calculation and Mel filtering: Compute the power spectrum of each frame and weighted by a triangular Mel filterbank to get the energy distribution on the Mel frequency scale; the output of the th Mel filter is ， wherein, is the frequency response function of the th Mel filter; S4.5, Mel frequency mapping: The linear frequencies are mapped to Mel frequencies using the following formula Mel(f) = 1, 25944 * log(l + f / 1000) ； S4.6, feature splicing and representation: The energies of all filter outputs are logarithmized and spliced to form a Mel spectrum, obtaining a high-dimensional acoustic feature representation of the keystroke segment as the input of the subsequent model.

6. The method of claim 1, wherein, The step S5 specifically includes self-supervised pre-training: S5.1, model construction: A network structure based on Masked Auto Encoder is constructed, the input is the frame-divided Mel spectrum and the mask strategy is adopted, only the unmasked patch is encoded; Two parallel linear prediction heads are set at the output end of the decoder, respectively for reconstruction and discrimination tasks; S5.2, reconstruction task: A reconstruction prediction head is set at the decoder end, and the recovery error of the masked patch is calculated by mean square error loss: ， wherein, is the reconstruction result of the th mask patch, is the original data, and M is the number of mask patches. S5.3, discrimination task: A discrimination prediction head is set in parallel, a contrast learning loss is introduced, and the model is constrained to aggregate the positive sample features of the same keystroke and separate the negative sample features of different keystrokes in the latent space, so as to learn a discriminative representation; S5.4, joint optimization: The total loss of pre-training is the weighted sum of reconstruction loss and discrimination loss: 。 7. The method of claim 6, wherein, The step S5.3, the loss function is: ， in, For the classification head prediction vector, It is the first An original patch is embedded.

8. The method of claim 1, wherein, The step S6 specifically includes fine-tuning and keystroke content restoration: S6.1, encoder migration: In the fine-tuning stage, the encoder structure and its weights obtained by pre-training are retained, used as a feature extractor, and the decoder is discarded, retaining the deep feature representation ability learned by the encoder during pre-training; S6.2, classification layer construction: A global pooling layer is added at the end of the encoder, and a fully connected classification head is connected, and finally a Softmax layer is connected, which aggregates the time sequence features of keystrokes into a global vector representation, realizes the classification of keystroke segments and the decoding of input content; S6.3, training and optimization: A smaller learning rate is used in the fine-tuning stage, a cross-entropy loss function is used for classification supervision, and an early stopping mechanism is introduced during training.

9. A computer terminal, characterized in that The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor.

10. A computer readable medium characterized by A computer program is stored thereon.

Images