Method, device and storage medium for recognizing multi-keyboard mixed key sound

Abstract

The application provides a multi-keyboard mixed key sound recognition method and device, equipment and a storage medium. The method comprises the following steps: acquiring a sound signal emitted when a keyboard is struck; performing keystroke signal interception on the sound signal to determine a keystroke signal segment; determining a mel-frequency cepstral coefficient according to the keystroke signal segment; inputting the mel-frequency cepstral coefficient into a preset single-key recognition model to output the corresponding input content of each keyboard. The scheme can be applied to the recognition of multi-keyboard mixed key sounds, and the recognition accuracy is high.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of signal recognition, and particularly relates to a multi-keyboard mixed key sound recognition method and device, equipment and a storage medium. BACKGROUND

[0002] The existing keyboard keystroke recognition mainly falls into two categories, one of which is to implant a malicious program on a computer to recognize the keyboard keystrokes, and the other of which is to recognize the keyboard keystroke content by using sound, WIFI, light and other signals. The second research method can be mainly divided into the following categories. (1) Based on WIFI signals, CSI technology is used to recognize the keyboard keystroke content, such as WiFinger, (2) based on light signals, video data is used to recognize the keyboard keystroke content, such as Blind Recognition of Touched Keys on Mobile Devices, (3) based on sound signals, the keyboard keystroke content is recognized, such as Accurate Combined Keystrokes Detection Using Acoustic Signals, which recognizes the keystroke combination (such as Ctrl+C) by capturing sound signals.

[0003] The existing keyboard keystroke recognition technology is to recognize a single key or a specific key combination (such as Ctrl+C) of a single keyboard, but in an office scenario, multiple keyboards are often struck at the same time, and the signal received by the recording device is often a mixed sound signal of multiple keyboards. Therefore, the existing keystroke sound recognition technology is not universal. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a multi-keyboard mixed key sound recognition method, device, equipment and storage medium.

[0005] To solve the above technical problems, the embodiments of the present application are implemented in the following way:

[0006] In a first aspect, the present application provides a multi-keyboard mixed key sound recognition method, which comprises the following steps:

[0007] Obtaining a sound signal emitted when a keyboard is struck;

[0008] Intercepting the keystroke signal from the sound signal to determine a keystroke signal segment;

[0009] Determining a mel-frequency cepstral coefficient according to the keystroke signal segment;

[0010] The mel-frequency cepstral coefficient input pre-set single key recognition model outputs the corresponding key input content of each keyboard.

[0011] In one of the embodiments, the sound signal generated by the keyboard stroke is acquired, including:

[0012] The sound signal generated by the keyboard stroke is acquired by the recording element of the terminal, and the terminal includes at least one recording element.

[0013] In one of the embodiments, the keystroke signal is intercepted from the sound signal to determine the keystroke signal segment, including:

[0014] The energy value of the signal segment in the sound signal is calculated every 41.7 ms;

[0015] If the energy value of the first signal segment is greater than the energy threshold value, the signal segment of the first preset time length before the starting point of the first signal segment and the second preset time length after the starting point is intercepted as the second signal segment;

[0016] The second signal segment uses the voice activity detection method to determine the keystroke signal segment.

[0017] In one of the embodiments, the second signal segment uses the voice activity detection method to determine the keystroke signal segment, including:

[0018] The second signal segment uses the voice activity detection method to determine the starting point and the ending point of the keystroke action, and extracts the keystroke signal;

[0019] The total energy, peak value and 5th wavelet transformed signal of the keystroke signal are calculated;

[0020] The total energy, peak value and 5th wavelet transformed signal of the keystroke signal are input into a pre-set support vector machine to determine whether the keystroke signal contains only one keystroke operation;

[0021] If the keystroke signal contains only one keystroke operation, a signal segment with a length of 41.7 ms is intercepted from the starting point to the rear as the keystroke signal segment;

[0022] If the keystroke signal contains two keystroke operations, a signal segment with a length of 41.7 ms is intercepted from the starting point to the rear as the first keystroke signal segment;

[0023] The starting position of the second keystroke operation is calculated by a regression neural network, and a signal segment with a length of 41.7 ms is intercepted from the starting position to the rear as the second keystroke signal segment;

[0024] The first keystroke signal segment and the second keystroke signal segment are used as the keystroke signal segment.

[0025] In one of the embodiments, the mel-frequency cepstral coefficients are determined according to the keystroke signal segment, comprising:

[0026] According to the keystroke signal segment, a low-pass filter is used for denoising to obtain a denoised signal segment;

[0027] The mel-frequency cepstral coefficients are determined according to the denoised signal segment.

[0028] In one of the embodiments, the preset single-key recognition model is constructed by the following steps:

[0029] The sound signal of the keystroke of each keyboard is obtained;

[0030] According to the sound signal, a voice activity detection method is used to cut the keystroke signal training segment with a duration of 41.7 ms from the sound signal;

[0031] A sound signal segment with the same length as the keystroke signal training segment is randomly obtained from the sound signal;

[0032] The sound signal segment and the keystroke signal training segment are superimposed to determine the noisy keystroke signal training segment;

[0033] The mel-frequency cepstral coefficient training set is determined according to all the keystroke signal training segments and all the noisy keystroke signal training segments, respectively;

[0034] The mel-frequency cepstral coefficient training set is used as input data to train the preset single-key recognition model.

[0035] In a second aspect, the present application provides a multi-keyboard mixed keystroke sound recognition device, which comprises:

[0036] The acquisition module is used to obtain the sound signal emitted during the keystroke of the keyboard;

[0037] The cutting module is used to cut the keystroke signal from the sound signal to determine the keystroke signal segment;

[0038] The determination module is used to determine the mel-frequency cepstral coefficients according to the keystroke signal segment;

[0039] The processing module is used to input the mel-frequency cepstral coefficients into the preset single-key recognition model and output the corresponding input content of each keyboard.

[0040] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-keyboard mixed keystroke sound recognition method of the first aspect.

[0041] In a fourth aspect, the present application provides a readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method for recognizing mixed key sounds of a multi-keyboard according to the first aspect.

[0042] From the technical solutions provided by the embodiments of the present specification, the solutions have the following advantages:

[0043] The method for recognizing mixed key sounds of a multi-keyboard provided by the embodiments of the present application can be applied to recognizing the input content of a multi-keyboard.

[0044] The method for recognizing mixed key sounds of a multi-keyboard provided by the embodiments of the present application only needs to use the recording elements on the terminal, without the need for additional equipment, and is low in cost and easy to obtain.

[0045] The method for recognizing mixed key sounds of a multi-keyboard provided by the embodiments of the present application proposes a BLSTM model based on an attention mechanism, uses the feature that there is a connection between the signals received by two recording elements in the same time period, and improves the accuracy of the BLSTM for key recognition to 96.41%. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0047] Figure 1 The flowchart of the method for recognizing mixed key sounds of a multi-keyboard provided by the present application is shown in the figure.

[0048] Figure 2 The experimental platform layout provided by the present application is shown in the figure.

[0049] Figure 3 The structure diagram of the preset single-key recognition model provided by the present application is shown in the figure.

[0050] Figure 4 The structure diagram of the device for recognizing mixed key sounds of a multi-keyboard provided by the present application is shown in the figure.

[0051] Figure 5 The structure diagram of the electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0052] In order to make the technical solutions in the specification better understood by the person skilled in the art, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the embodiments of the specification. Obviously, the described embodiments are only part of the embodiments of the specification, rather than all the embodiments. Based on the embodiments in the specification, all other embodiments obtained by the person skilled in the art without creative labor should belong to the protection scope of the specification.

[0053] In the following description, specific details are set forth in connection with the particular structures, techniques, etc., in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to one skilled in the art that the application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the application.

[0054] Many modifications and variations of the specific implementation of the application described in the specification can be made without departing from the scope or spirit of the application, which will be apparent to one skilled in the art. Other implementations of the application will be apparent to those skilled in the art from the specification. The specification and embodiments are merely exemplary.

[0055] As used herein, "include," "includes," "including," "have," "has," "having," or the like are open-ended terms that are intended to mean "including but not limited to."

[0056] The "parts" in the present application are measured by mass unless otherwise specified.

[0057] The application will be further described in detail below with reference to the accompanying drawings and examples.

[0058] Referring to Figure 1 , which shows a flowchart of a multi-keyboard mixed key sound recognition method provided by the embodiments of the application.

[0059] As Figure 1 shown, the multi-keyboard mixed key sound recognition method can include:

[0060] S110, acquiring a sound signal emitted when a keyboard is struck.

[0061] Specifically, the sound signal emitted when the keyboard is struck is collected by the recording element of the terminal, and the collected sound signal is uploaded to the cloud. The terminal can include any electronic device with a recording element, such as a mobile phone, a tablet computer, a wearable device, etc. The recording element can be a microphone. The terminal can include at least one microphone, for example, a mobile phone can include two or more microphones.

[0062] As shown in Figure 2 When the sound signal emitted when the keyboard is struck is collected by the recording element of the mobile phone, the mobile phone is placed in the middle of the two keyboards, and the two or more recording elements on the mobile phone collect the keystroke sound and upload it to the cloud.

[0063] S120, keystroke signal interception is performed on the sound signal to determine the keystroke signal segment.

[0064] Specifically, in the cloud, the collected sound signal is subjected to keystroke signal interception, and then the intercepted keystroke signal is subjected to signal segment cutting to obtain the keystroke signal segment. It can be understood that if the keystroke signal only contains one keystroke operation, one keystroke signal segment can be obtained, and if the keystroke signal contains two keystroke operations, two keystroke signal segments can be obtained.

[0065] In one embodiment, S120, keystroke signal interception is performed on the sound signal to determine the keystroke signal segment, which can include:

[0066] The energy value of the signal segment in the sound signal is calculated every 41.7 ms;

[0067] If the energy value of the first signal segment is greater than the energy threshold value, the signal segment of the first preset time length before the starting point of the first signal segment and the second preset time length after the starting point is intercepted as the second signal segment;

[0068] The second signal segment uses a voice activity detection method to determine the keystroke signal segment.

[0069] Among them, the second signal segment uses a voice activity detection method to determine the keystroke signal segment, which can include:

[0070] The second signal segment uses a voice activity detection method to determine the starting point and the ending point of the keystroke action, and extracts the keystroke signal;

[0071] The total energy, peak value and 5th wavelet transformed signal of the keystroke signal are calculated;

[0072] The total energy, peak value and 5th wavelet transformed signal of the keystroke signal are input into a preset support vector machine to determine whether the keystroke signal contains only one keystroke operation;

[0073] If the keystroke signal contains only one keystroke operation, a signal segment with a length of 41.7 ms is intercepted from the starting point as the keystroke signal segment;

[0074] If the keystroke signal contains two keystroke operations, a signal segment with a length of 41.7 ms is intercepted from the starting point as the first keystroke signal segment;

[0075] The starting position of the second keystroke operation is calculated by a regression neural network, and a signal segment with a length of 41.7 ms is cut from the starting position backward as a second keystroke signal segment;

[0076] The first keystroke signal segment and the second keystroke signal segment are keystroke signal segments.

[0077] Specifically, the energy threshold value can be set according to actual needs. The first preset time length and the second preset time length can be set according to actual needs, for example, the first preset time length and the second preset time length are both 1 s.

[0078] The energy value of the signal segment is: a

[0079] wherein, n is the length of the signal segment.

[0080] For the received sound signal, the energy value of the signal segment is calculated every 41.7 ms, and if the energy value exceeds the threshold value, the signal segment (i.e. the first signal segment) is cut from the starting point to the signal segment (i.e. the second signal segment) for 1 s before and after (the total duration is 2 s) as a signal segment (i.e. the second signal segment) that may exist keystroke action.

[0081] For the cut signal segment, the starting point stp and the ending point of the keystroke action are found using the Voice Activity Detection (VAD) method, and the keystroke signal is extracted.

[0082] For the keystroke signal extracted by VAD, the total energy value, kurtosis and 5th wavelet transform signal are calculated, and whether the keystroke signal contains only one keystroke operation is judged by the trained SVM (Support Vector Machine). If the keystroke signal contains only one keystroke operation, a signal segment with a length of 41.7 ms is cut from the starting point stp backward as a keystroke signal segment, and the mel frequency cepstral coefficient is determined according to the keystroke signal segment in step S130. If the keystroke signal contains two keystroke operations, the first keystroke signal segment is a signal segment with a length of 41.7 ms cut from the starting point stp backward, and then the position inv (i.e. the time when the two keystroke operations start to overlap) of the start of the second keystroke operation is calculated by a regression neural network. The second keystroke signal segment is a signal segment with a length of 41.7 ms cut from inv backward, and the mel frequency cepstral coefficient is determined according to the first keystroke signal segment and the second keystroke signal segment in step S130.

[0083] The specific operation of obtaining the regression neural network model for calculating the overlap starting position is as follows:

[0084] ​The application adopts an LSTM-based regression neural network model to calculate the overlapping start position, and the network structure thereof comprises an input layer, an LSTM layer, a Flatten layer and a dense (full connection) layer. The model generates an overlapping signal containing two keystroke operations by randomly superimposing (random overlapping start position, signal source and label) single key signals in the keystroke signal segment set, while recording the overlapping start position as a label.

[0085] The input layer receives the intercepted keystroke signal segment as the input of the model.

[0086] The LSTM layer encodes the input data of the model, so that the output data of the LSTM contains time sequence information.

[0087] The Flatten layer changes the output data of the LSTM layer into a one-dimensional vector, facilitating the calculation of the full connection layer.

[0088] The full connection layer multiplies the input data of the full connection layer with the weight value to obtain the estimated overlapping start position. The layer does not use an activation function.

[0089] The loss function is set as

[0090] .

[0091] It can be understood that, for the keystroke signal segment, the application determines the signal source (i.e., which keyboard the keystroke signal comes from) by calculating the energy difference of the signal segments received by different recording elements.

[0092] The specific operation of determining the signal source is as follows:

[0093] (1) The keystroke signal segments received by the two recording elements are aligned in time.

[0094] (2) After alignment, the total energy values of the signal segments of the two recording elements are calculated respectively, and the difference is obtained.

[0095] (3) Since the path lengths of the same sound source reaching the two recording elements are different, the degree of attenuation of the keystroke signal is also different. The longer the path, the higher the degree of attenuation, that is, the lower the total energy of the signal received by the recording element. The two keyboards are located on the two sides of the two recording elements, so the total energy difference corresponding to one keyboard is always positive, and the total energy difference corresponding to the other keyboard is always negative. Thus, the keystroke signal source can be determined.

[0096] S130, determining the mel frequency cepstral coefficient according to the keystroke signal segment, can comprise:

[0097] According to the keystroke signal segment, a low-pass filter is used for denoising to obtain a denoised signal segment;

[0098] According to the denoised signal segment, a mel-frequency cepstral coefficient is determined.

[0099] For the keystroke signal segment, a low-pass filter is used for denoising to obtain a denoised signal segment; and according to the denoised signal segment, a mel-frequency cepstral coefficient is calculated as input data of a preset single-key recognition model.

[0100] In the field of sound, the mel-frequency cepstrum is a linear transformation of the log-energy spectrum of the sound frequency based on the nonlinear mel scale, and the mel-frequency cepstral coefficient is a coefficient that constitutes the mel-frequency cepstrum. It takes into account the human auditory characteristics, first maps the linear spectrum to the mel nonlinear spectrum based on auditory perception, and then converts to the cepstrum.

[0101] The specific operation of calculating the mel-frequency cepstral coefficient is as follows:

[0102] (1) The denoised signal segment is pre-emphasized, framed and windowed.

[0103] (2) For each frame, the corresponding frequency spectrum is obtained by FFT (Fast Fourier Transform).

[0104] (3) For the obtained frequency spectrum, the mel-frequency spectrum is obtained by the mel filter bank.

[0105] (4) The mel-frequency cepstral coefficient is obtained by performing log, inverse transform and other cepstrum analysis operations on the mel-frequency spectrum.

[0106] S140, the mel-frequency cepstral coefficient is input into a preset single-key recognition model, and the corresponding input content of each keyboard is output.

[0107] Specifically, the preset single-key recognition model can be pre-trained. The preset single-key recognition model is a BLSTM neural network model (Bidirectional Long Short-Term Memory Recurrent Neural Network) based on attention mechanism, and the network structure is as shown in Figure 3 It includes two input layers, two BLSTM layers, one concatenate layer, one attention layer and one dense layer.

[0108] Input layer: Since there is some relationship between the signals received by the two recording elements in the same time period, this neural network uses two input layers to receive the mel-frequency cepstral coefficients corresponding to the two recording elements as its input data.

[0109] BLSTM layer: BLSTM is composed of forward LSTM (Long Short Term Memory recurrent neural network) and backward LSTM, which is often used to model context information in natural language processing tasks, that is, the data obtained by processing the time series data through BLSTM contains forward and backward information. Therefore, the application adopts two BLSTM layers to receive the output data from the two input layers respectively, and encodes the input layer data, so that the output sequence of BLSTM contains time series information.

[0110] concatenate layer: concatenate the output sequences of the two BLSTM layers.

[0111] attention layer: process the sequence after concatenating the two BLSTM, so that the attention output data contains the association information between the signals received by the two recording elements.

[0112] dense layer: full connection layer, process the data output by the attention layer to obtain the result of key recognition. The activation function used in the full connection layer of the application is sigmoid function, and the output dimension is set to the number of labels.

[0113] In one embodiment, the preset single key recognition model is constructed by the following steps:

[0114] obtain the sound signal of each key stroke of the keyboard;

[0115] According to the sound signal, use the voice activity detection method to cut the keystroke signal training segment with a duration of 41.7ms from the sound signal;

[0116] randomly obtain a sound signal segment with the same length as the keystroke signal training segment from the sound signal;

[0117] superimpose the sound signal segment and the keystroke signal training segment to determine the noisy keystroke signal training segment;

[0118] determine the mel frequency cepstral coefficient training set according to all keystroke signal training segments and all noisy keystroke signal training segments respectively;

[0119] The mel frequency cepstral coefficient training set is used as input data to train the preset single key recognition model.

[0120] Specifically, the sound signals of key strokes are collected by using a commercial mobile phone, and the user is required to strike the keys every 2 seconds to avoid overlapping of the key stroke signals; for the collected sound signals, the sound signals every two seconds are taken as a group of signals to be processed; for the signals to be processed every two seconds, a VAD (Voice Activity Detection) algorithm is used to intercept a key stroke signal segment with a duration of about 41.7 ms from the signals; for each intercepted key stroke signal segment, an equal-length sound signal segment collected is randomly superimposed as a key stroke signal segment after adding noise, so as to increase the data quantity of the training set; for the obtained key stroke signal segment set, a trained support vector machine model is obtained through the total energy value, kurtosis and the signal after 5 times of wavelet transform, and is used to judge whether the key stroke signal segment contains only one key stroke operation; for the intercepted key stroke signal segment and the key stroke signal segment after adding noise, the mel-frequency cepstral coefficients are calculated to generate a training set; and for the training set, the single-key recognition model is trained and obtained by taking the training set as input data.

[0121] In order to cut out the key stroke signal from the original signal and avoid classification of data not containing the key stroke signal by the background, a common voice activity detection algorithm, double-threshold endpoint detection method, is used to identify and eliminate long silent periods.

[0122] The specific operation of intercepting the key stroke signal segment by using the VAD algorithm is as follows:

[0123] (1) For the original signal, the formula is used for normalization to obtain a normalized signal, wherein L is the length of the original signal.

[0124] The normalized signal is substituted into the formula for updating, so as to introduce timing information.

[0125] (2) For the signal after introducing the timing information, the total energy of a signal with a length of FrameLen is calculated every FrameInc sampling points, and an array amp is obtained, which is a set of total energies of each frame. Specifically, a signal with a length of FrameLen is extracted as a frame i with FrameInc as the step, and the sum of absolute values of the frame is calculated as the total energy of the frame, that is, amp[i].

[0126] (3) A higher short-time energy threshold MH and a lower short-time energy threshold ML are calculated according to the maximum value of amp. Wherein, MH = min(max(amp) / 4, 10); ML = min(max(amp) / 8, 2). If amp[i] > ML, the frame is likely to be in the pronunciation stage (the frame is set as status1), and when the number of status1 frames is more than 15, it is considered that the pronunciation stage is determined to be entered.

[0127] (4) Calculate the short-time zero-crossing rate (i.e., the number of times crossing the horizontal axis of the coordinate axis per unit time). Wherein, the short-time zero-crossing rate is calculated for each frame respectively, and an array zcr is obtained. Specifically, zcr[i] is the number of times crossing the horizontal axis of the coordinate axis in frame i divided by the frame length FrameLen.

[0128] (5) Traverse the array amp, if amp[i] exceeds the threshold value MH, the first reference starting point stp1 is obtained.

[0129] (6) Traverse backward from stp1, if amp[i] exceeds the threshold value MH or the short-time zero-crossing rate zcr[i] exceeds the threshold value Zs, it is considered that the keystroke sound is still continuing, and the traversal is continued backward, otherwise, it is considered that the keystroke sound is ended. Wherein, the threshold value Zs can be set according to actual requirements.

[0130] (7) According to the starting point and the ending point found, the keystroke signal segment is intercepted.

[0131] The specific operation of obtaining the support vector machine model for judging whether the keystroke signal segment contains only one keystroke operation is as follows:

[0132] (1) Use the single key signal in the keystroke signal segment set to generate an overlapping signal containing two keystroke operations (the overlapping starting position, the signal source and the label are random). Wherein, the overlapping starting position (a numerical value) is randomly generated; two keystroke signals are randomly selected (the signal source and the key from which the signal comes are random); and the selected keystroke signals are linearly overlapped according to the generated overlapping starting position.

[0133] (2) Label the single key signal and the generated overlapping signal of the training set to generate the original data of the support vector machine training set.

[0134] (3) The difference between the double key signal and the single key signal is as follows: a. The double key signal generally presents three or more peaks in the time domain; b. The total energy of the double key signal is higher than that of the original single key signal; c. A large energy hit peak appears in the second half of the keystroke signal. Therefore, the total energy value and the kurtosis (peak degree) of the received keystroke signal segment are extracted as the judgment features for distinguishing whether the keystroke signal segment contains a double key signal. At the same time, in order to describe the difference in the number of peaks between double key and single key signals and reduce the amount of training data, the signal after 5 times of wavelet transform of the original signal is used as the judgment feature. Therefore, the total energy value, the kurtosis and the signal after 5 times of wavelet transform of the original data of the training set are calculated as the input features of the support vector machine, and the training set for judging whether it contains only a single keystroke operation is generated.

[0135] (4) Train and obtain the SVM model through the training set.

[0136] The formula for calculating the total energy value is: wherein, n is the length of the signal segment.

[0137] The formula for calculating the kurtosis is: .

[0138] The multi-keyboard mixed key sound recognition method provided by the embodiments of the present application can be applied to recognizing the input content of a multi-keyboard.

[0139] The multi-keyboard mixed key sound recognition method provided by the embodiments of the present application only needs to use the recording element on the terminal, without the need for additional equipment, low cost and easy to obtain.

[0140] The multi-keyboard mixed key sound recognition method provided by the embodiments of the present application proposes a BLSTM model based on an attention mechanism, uses the feature that there is a connection between the signals received by two recording elements in the same time period, and improves the accuracy of key recognition by BLSTM to 96.41%.

[0141] Experimental verification

[0142] Experimental environment: The experiments were conducted in a conference room and a dormitory. The conference room environment is relatively quiet, and the noise mainly comes from the sound of passing vehicles in the distance, the sound of the air conditioner, and the sound of key sound reflection. There are many objects in the conference room, and the environment is relatively complex. The dormitory environment is relatively noisy, with various human voices, non-target keyboard keystrokes, and washing machine sounds, and a series of interference noises, which bring challenges to the extraction of keystroke signal segments. At the same time, there are more objects in the dormitory, and the environment is more complex, resulting in more complex sound reflected back by the key sound. In order to avoid the influence of the material of the table under the keyboard and the vibration of the table when the key is struck, the keyboard and the mobile phone are placed on the mouse pad, and the keyboard and the mobile phone are fixed on the mouse pad to avoid slight changes in the position of the keyboard during the process of striking the keyboard.

[0143] Keyboard: The experiment is mainly conducted on a mechanical keyboard. The mechanical keyboard model is iKBC typeman W200, and the mechanical keyboard has not been used before collecting data, and there is no key wear. The keystroke sound of the mechanical keyboard is relatively clear, and the key position is stable. The duration of the complete single key signal is about 125 ms, and the duration of the hit peak is about 42 ms.

[0144] Mobile phone: The software is deployed on Huawei P20 and Redmi K30 mobile phone platforms respectively to collect button sound, transmit data and display eavesdropped text. Huawei P20 has 2 microphones, one on the top and one on the bottom of the mobile phone, and uses Android 8.1 system, providing a maximum sampling rate of 48 kHz. Redmi K30 has 3 microphones, one on the top, one on the bottom and one in the middle of the four cameras, and uses Android 10.0 system, providing a maximum sampling rate of 96 kHz. When the software is deployed on the Redmi K30 mobile phone platform, it can only call the two microphones on the top and bottom. Therefore, the data collected on the two mobile phone platforms are all double-channel data, with a sampling rate of 48 kHz for Huawei P20 and a sampling rate of 96 kHz for Redmi.

[0145] Knocking speed: The tester is required to knock the button every 2 seconds to avoid overlapping signals received by the microphone.

[0146] Dataset: The tester is required to knock the buttons A to Z, a total of 26 buttons, 60 times for each button. In order to rule out the possibility that this application regards the stable features changing over time in the environment (such as the sound of people talking and the sound of outdoor music playing) as the features of button classification, the tester is required to divide the 60 times of each button into 3 times, collect 20 groups of audio signals of button knocking each time, and the time interval of each time is at least 4 hours.

[0147] Single key recognition effect

[0148] The recognition accuracy of 26 buttons on a keyboard can reach 96.41% at most.

[0149] Double key recognition effect

[0150] The recognition accuracy of the mixed signal of two keyboards can reach 67% at most.

[0151] Overall simulation experiment: The single key signals of two keyboards are linearly superimposed, and the starting position of the overlap is a randomly generated value inv. The linearly superimposed signal is used to simulate the mixed signal of multiple keyboards. The overlap starting point, signal source and label of the mixed signal are randomly selected.

[0152] Effect of signal source judgment

[0153] Premise: The starting position of the overlap is known

[0154] Single key judgment accuracy: 99.87%

[0155] Double key judgment accuracy: 94.37%

[0156] Double key recognition effect

[0157] Premise: the overlapping start position and signal source are known

[0158] The recognition accuracy of the first key is 83.25%.

[0159] The recognition accuracy of the second key is 74.84%.

[0160] Reference Figure 4 It shows the structure diagram of the multi-keyboard mixed key sound recognition device described according to an embodiment of the application.

[0161] As Figure 4 shown, the multi-keyboard mixed key sound recognition device 400 can include:

[0162] The acquisition module 410 is configured to acquire the sound signal emitted when the keyboard is struck.

[0163] The interception module 420 is configured to intercept the keystroke signal from the sound signal and determine the keystroke signal segment.

[0164] The determination module 430 is configured to determine the mel-frequency cepstral coefficient based on the keystroke signal segment.

[0165] The processing module 440 is configured to input the mel-frequency cepstral coefficient into a preset single-key recognition model and output the corresponding input content of each keyboard.

[0166] Optionally, the acquisition module 410 is further configured to:

[0167] acquire the sound signal emitted when the keyboard is struck sent by the recording element of the terminal, and the terminal includes at least one recording element.

[0168] Optionally, the interception module 420 is further configured to:

[0169] calculate the energy value of the signal segment in the sound signal every 41.7 ms;

[0170] if the energy value of the first signal segment is greater than the energy threshold value, intercept the signal segment of the first preset time length before the start point of the first signal segment and the second preset time length after the start point, as the second signal segment;

[0171] The second signal segment is determined by using a voice activity detection method.

[0172] Optionally, the interception module 420 is further configured to:

[0173] The second signal segment is determined by using a voice activity detection method to determine the start point and the end point of the keystroke action, and the keystroke signal is extracted.

[0174] The total energy, peak value, and signal after 5 times of wavelet transform of the keystroke signal are calculated.

[0175] input the total energy, the peak value and the signal after 5 times wavelet transform of the keystroke signal into the preset support vector machine to determine whether the keystroke signal only contains one keystroke operation;

[0176] If the keystroke signal only contains one keystroke operation, a signal segment with a length of 41.7 ms is intercepted from the starting point to the rear as a keystroke signal segment;

[0177] If the keystroke signal contains two keystroke operations, a signal segment with a length of 41.7 ms is intercepted from the starting point to the rear as a first keystroke signal segment;

[0178] The starting position of the beginning of the second keystroke operation is calculated through the regression neural network, a signal segment with a length of 41.7 ms is intercepted from the starting position to the rear as a second keystroke signal segment;

[0179] The first keystroke signal segment and the second keystroke signal segment are used as the keystroke signal segment.

[0180] Optionally, the determination module 430 is further configured to:

[0181] According to the keystroke signal segment, a low-pass filter is used for denoising to obtain a denoised signal segment;

[0182] According to the denoised signal segment, a mel-frequency cepstral coefficient is determined.

[0183] Optionally, the processing module 440 is further configured to:

[0184] Obtain a sound signal when each key of the keyboard is struck;

[0185] According to the sound signal, a voice activity detection method is used to intercept a keystroke signal training segment with a duration of 41.7 ms from the sound signal;

[0186] Randomly obtain a sound signal segment with the same length as the keystroke signal training segment from the sound signal;

[0187] The sound signal segment and the keystroke signal training segment are superimposed to determine a noisy keystroke signal training segment;

[0188] According to all keystroke signal training segments and all noisy keystroke signal training segments, a mel-frequency cepstral coefficient training set is determined;

[0189] The mel-frequency cepstral coefficient training set is used as input data to train a preset single-key recognition model.

[0190] The multi-keyboard mixed keystroke sound recognition device provided in the embodiment can execute the embodiments of the above method, and has similar implementation principles and technical effects, which will not be described here.

[0191] Figure 5 A structural diagram of an electronic device according to an embodiment of the present application is provided. As shown in Figure 5 FIG. 1, a structural diagram of an electronic device 300 suitable for implementing embodiments of the present application is shown.

[0192] As shown in Figure 5 FIG. 1, the electronic device 300 includes a central processing unit (CPU) 301 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or programs loaded from a storage section 308 into a random access memory (RAM) 303. Various programs and data required for the operation of the device 300 are also stored in the RAM 303. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0193] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 306 as necessary. A removable media 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 310 as necessary, so that a computer program read therefrom is installed in the storage section 308 as necessary.

[0194] In particular, the processes described above with reference to Figure 1 the present disclosure can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for executing the above-described method of recognizing a multi-keyboard mixed key sound. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from the removable media 311.

[0195] The computer program product of the present application can be a computer program embodied on a non-transitory computer readable medium. When the program is executed by a computer, it implements the functions of the present application. The computer program product can be a floppy disk, a USB flash drive, a DVD, a CD, a memory stick, or a computer memory. The computer program product can also be an electronic file that is downloaded from a network, such as the Internet, and that contains the computer program.

[0196] The units or modules described in the embodiments of the present application can be implemented by software, or by hardware. The units or modules described can also be implemented in a processor. The names of the units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0197] The system, device, module or unit described in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop, a mobile phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0198] As another aspect, the present application also provides a storage medium, which can be a storage medium contained in the foregoing device in the above embodiments; or can exist independently, without being assembled into a device. The storage medium stores one or more programs, which are used by one or more processors to implement the method for recognizing mixed key sounds of a multi-keyboard.

[0199] Storage media includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition herein, computer readable medium does not include transitory computer readable media, such as modulated data signals and carrier waves.

[0200] It should be noted that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0201] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

Claims

1. A method for recognizing a mixed key sound of a plurality of keyboards, characterized by, The method comprises: acquiring a sound signal emitted when a keyboard is struck, comprising: acquiring a sound signal emitted when a keyboard is struck by a recording element of a terminal, the terminal comprising at least one recording element; performing keystroke signal interception on the sound signal to determine a keystroke signal segment, comprising: calculating an energy value of a signal segment in the sound signal every 41.7 ms; if the energy value of a first signal segment is greater than an energy threshold value, intercepting a signal segment of a first preset time length before the start point of the first signal segment and a second preset time length after the start point of the first signal segment as a second signal segment; determining the keystroke signal segment by using a voice activity detection method on the second signal segment, comprising: determining the start point and the end point of a keystroke action by using the voice activity detection method on the second signal segment, and extracting a keystroke signal; calculating the total energy, the peak value, and the signal after 5 times of wavelet transform of the keystroke signal; inputting the total energy, the peak value, and the signal after 5 times of wavelet transform of the keystroke signal into a preset support vector machine to determine whether the keystroke signal only contains one keystroke operation; if the keystroke signal only contains one keystroke operation, intercepting a signal segment of a length of 41.7 ms starting from the start point as the keystroke signal segment; if the keystroke signal contains two keystroke operations, intercepting a signal segment of a length of 41.7 ms starting from the start point as a first keystroke signal segment; calculating the start position of the occurrence of a second keystroke operation by a regression neural network, and intercepting a signal segment of a length of 41.7 ms starting from the start position as a second keystroke signal segment; the first keystroke signal segment and the second keystroke signal segment are the keystroke signal segment; determining a mel-frequency cepstral coefficient according to the keystroke signal segment; inputting the mel-frequency cepstral coefficient into a preset single-key recognition model to output the corresponding input content of each keyboard.

2. The method of claim 1, wherein, The method of determining a mel-frequency cepstral coefficient according to the keystroke signal segment comprises: adopting a low-pass filter to denoise the keystroke signal segment to obtain a denoised signal segment; determining the mel-frequency cepstral coefficient according to the denoised signal segment.

3. The method of claim 1, wherein, The preset single-key recognition model is constructed by the following steps: acquiring a sound signal emitted when a key of each keyboard is struck; adopting a voice activity detection method to intercept a keystroke signal training segment with a duration of 41.7 ms from the sound signal according to the sound signal; randomly acquiring a sound signal segment with the same length as the keystroke signal training segment from the sound signal; superimposing the sound signal segment and the keystroke signal training segment to determine a noisy keystroke signal training segment; determining a mel-frequency cepstral coefficient training set according to all the keystroke signal training segments and all the noisy keystroke signal training segments respectively; using the mel-frequency cepstral coefficient training set as input data to train the preset single-key recognition model.

4. A multi-keyboard mixed key sound recognition apparatus, characterized by comprising: The device comprises: The acquisition module is configured to acquire the sound signal emitted by the keyboard strike, including: acquiring the sound signal emitted by the keyboard strike sent by the recording element of the terminal, wherein the terminal includes at least one recording element; The interception module is configured to intercept the keystroke signal from the sound signal and determine the keystroke signal segment, including: calculating the energy value of the signal segment in the sound signal every 41.7 ms; if the energy value of the first signal segment is greater than the energy threshold value, intercepting the signal segment of the first preset time length before the starting point of the first signal segment and the second preset time length after the starting point of the first signal segment as the second signal segment; using the voice activity detection method to determine the keystroke signal segment from the second signal segment, including: using the voice activity detection method to determine the starting point and the ending point of the keystroke action from the second signal segment, and extracting the keystroke signal; calculating the total energy, the peak value, and the signal after 5 times of wavelet transform of the keystroke signal; inputting the total energy, the peak value, and the signal after 5 times of wavelet transform of the keystroke signal into a preset support vector machine to determine whether the keystroke signal only includes one keystroke operation; if the keystroke signal only includes one keystroke operation, intercepting the signal segment with a length of 41.7 ms from the starting point backward as the keystroke signal segment; if the keystroke signal includes two keystroke operations, intercepting the signal segment with a length of 41.7 ms from the starting point backward as the first keystroke signal segment; calculating the starting position of the occurrence of the second keystroke operation from the starting position by using a regression neural network, and intercepting the signal segment with a length of 41.7 ms from the starting position backward as the second keystroke signal segment; The first keystroke signal segment and the second keystroke signal segment are used as the keystroke signal segment; The determination module is configured to determine the mel-frequency cepstral coefficient according to the keystroke signal segment; The processing module is configured to input the mel-frequency cepstral coefficient into a preset single-key recognition model, and output the corresponding input content of each keyboard.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the multi-keyboard mixed key sound recognition method according to any one of claims 1-3.

6. A readable storage medium, having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the multi-keyboard mixed key sound recognition method according to any one of claims 1-3.

Images