A method, device, computer equipment and storage medium for identifying keystroke content based on audio

Abstract

The present invention relates to a method, device, equipment and storage medium for identifying keystroke content based on audio, belonging to the field of wireless identification, the method comprising: receiving audio during keystrokes; locating keystroke segments in the audio; generating a keystroke constraint set using a keystroke relative position prediction method; selecting a constraint subset in the keystroke constraint set; and restoring the keystroke content in combination with a corpus. The present application does not require the use of particularly high-precision equipment, which saves costs and improves applicability; in addition, the use of multiple positioning methods to generate keystroke constraints avoids the one-sidedness of a single method; finally, the identification method does not rely on specific scenarios and equipment, and even if different keyboards are used, different keyboard users, etc., it will not affect the identification, thereby improving the accuracy of keystroke content recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless identification, and in particular to a method, device, computer equipment and storage medium for identifying keystroke content based on audio. Background Art

[0002] When you type on the keyboard, there is usually some related information, such as the sound produced when typing on the keyboard, the radiation emitted by the keyboard, the interval between the keys, etc. By studying this information, the content of the keystrokes can be identified through these seemingly insignificant information. For example, researchers use the TDOA (time difference of arrival) information of the sound of typing on the keyboard to different microphones to reconstruct the shape and position of the keyboard, and continuously narrow the "hot spot" area through multiple smartphone devices, and finally accurately locate it on a key. Some researchers have conducted electromagnetic leakage analysis on PS / 2 interface keyboards to restore the keys; some have used the accelerometer built into the iPhone to obtain relevant acceleration information, judge whether the tapping point is on the left or right according to the distance of the keyboard tapping, and guess the content of the keyboard tapping in combination with the English dictionary.

[0003] In existing applications, many keystroke content recognition methods rely on machine learning methods, which require a large amount of data as support. However, each person often has different keystroke habits when typing on the keyboard, which leads to insufficient generalization of these machine learning models. The use of these machine learning models often greatly reduces the universality of these recognition methods. In addition, some recognition methods require the use of high-precision equipment, which makes the scope of use too narrow and too dependent on the equipment. The accuracy of recognition is greatly affected by the data collection method. Finally, existing applications usually only target specific conditions, such as specific external scenes, specific keyboards, etc. Once the external conditions change, the recognition accuracy will be low. Summary of the invention

[0004] Based on the problems existing in the prior art, the present application provides an audio-based keystroke content recognition method, apparatus, computer equipment and storage medium, which generates a keystroke constraint set by combining the TDOA method and the relevance sorting method, selects a corresponding constraint subset from the keystroke constraint set, and finally restores the keystroke content from the constraint subset in combination with the corpus. This method does not rely on specific scenarios and equipment, and will not affect the recognition even if different keyboards are used or different users of the keyboards are used, thereby improving the accuracy of keystroke content recognition.

[0005] The first aspect of the present application provides a method for identifying keystroke content based on audio, comprising:

[0006] Receive audio from keystrokes;

[0007] Locate keystroke fragments in audio;

[0008] Determine one or more keystroke constraint division methods according to keyboard arrangement rules;

[0009] Generating a keystroke constraint set for the located keystroke fragments using a keystroke relative position prediction method corresponding to one or more keystroke constraint partitioning methods, wherein the keystroke relative position prediction method includes a TDOA method and a relevance ranking method;

[0010] selecting a constraint subset from the keystroke constraint set;

[0011] The keystroke content is restored from the constraint subset in combination with the corpus.

[0012] A second aspect of the present application provides an audio-based keystroke content recognition device, comprising:

[0013] An audio receiving module, used to receive audio when typing keys;

[0014] The original audio of keystrokes is collected through a microphone. When collecting the audio of keystrokes, multiple microphones are needed to collect multiple audio data for the same keystroke sequence.

[0015] An audio processing module for locating keystroke segments in audio;

[0016] Locating the keystroke segment in the audio mainly refers to finding the start and end sampling points (or time) of the keystroke segment and the release segment in the audio.

[0017] A constraint division module, used to determine one or more keystroke constraint division methods according to keyboard arrangement rules;

[0018] A constraint generation module, configured to generate a keystroke constraint set for the located keystroke fragments using a keystroke relative position prediction method corresponding to one or more keystroke constraint division methods, wherein the keystroke relative position prediction method includes a TDOA method and a relevance ranking method;

[0019] In the present invention, the relative position of keystrokes mainly refers to the relationship between the positions of each keystroke, and the relationship between these keystroke positions is called a constraint. Different relative position prediction methods can be selected to correspond to different keystroke constraint division methods to generate a keystroke constraint set.

[0020] A constraint selection module, configured to select a constraint subset from the keystroke constraint set;

[0021] Since the constraints in the constraint set are not completely correct, there will be wrong constraints. If all constraints are selected to guess the keystroke content, there will often be contradictory constraint subsets.

[0022] Therefore, it is necessary to avoid conflicting constraints when selecting. At the same time, since even a single wrong constraint may lead to a large error in the subsequent results, only some constraints need to be selected from the keystroke constraint set as the constraint subset for subsequent recognition of keystroke content.

[0023] Optionally, set the ratio to select a portion of non-contradictory constraints as the keystroke constraints used to finally restore the keystroke content.

[0024] The content identification module is used to restore the keystroke content from the constraint subset in combination with the corpus.

[0025] The third aspect of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the first aspect of the present application.

[0026] A fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, which, when executed by a processor, cause the processor to execute the method in the first aspect.

[0027] According to the first aspect of the present application, the fifth aspect of the present application further provides a computer program product or a computer program, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional modes such as the above-mentioned first aspect.

[0028] Beneficial effects of the present invention:

[0029] The present application receives audio of keystrokes; locates the keystroke segments in the audio; selects a plurality of keystroke relative position prediction methods to generate a keystroke constraint set; selects a constraint subset in the keystroke constraint set; and restores the keystroke content in combination with a corpus. This keystroke prediction method does not need to rely on a data set, which improves the universal applicability of the recognition method. In addition, the recognition method does not rely on specific scenarios and devices, and will not affect the recognition even if different keyboards are used or different users of the keyboards are used.

[0030] Normally, when using the TDOA method for keystroke prediction, due to the tight arrangement of the keyboard, it is usually necessary to use multiple microphones for integration, and gradually narrow the hot spot area for positioning. This keystroke prediction method is highly dependent on the number of microphones and the sampling frequency. However, the present application mainly uses the TDOA method to predict keystroke constraints, so only the approximate position of the keystroke is required, and multiple microphones are not required to determine the hot spot area, which reduces the requirements for microphone equipment. The present application combines the TDOA method and the correlation sorting method when performing keystroke constraints. Since the traditional correlation sorting can only predict the relative position relationship between the keys, but cannot determine the absolute keystroke position, the use of the TDOA method makes up for this shortcoming, and the approximate area where the keys are located can be determined, thereby improving the accuracy of the key constraints. The correlation sorting method also makes up for the situation that when the number of microphones is small, the number of keys crossed by the TDOA hyperbola is too large, which makes it difficult to determine two adjacent keys or the same key, thereby improving the accuracy of the constraint generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of a method for identifying keystroke content based on audio provided in an embodiment of the present application;

[0032] Figure 2 It is a schematic diagram of dividing keystroke segments in the time domain provided by the present application;

[0033] Figure 3 It is a schematic diagram of an audio-based keystroke content recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The present application relates to the relevant technology in the field of wireless identification, and the keystroke content is identified by a non-contact method. At the same time, the present application relates to the relevant technology in the field of acoustics, wherein acoustics refers to the science of studying the generation, propagation, reception and effect of mechanical waves. Acoustics is one of the earliest branches of physics to be studied in depth. With the invention and application of radio technology in the 19th century, the generation, transmission, reception and measurement technology of mechanical waves have made leaps and bounds. This acoustics has entered the development period of modern acoustics from ancient classical acoustics. The permeability of modern acoustics is very strong. Acoustics intersects with many other disciplines (such as physics, chemistry, materials, life, geology, environment, etc.), engineering technology (such as machinery, architecture, electronics, communications, etc.) and art fields, and has played an important and unique role in these fields, and further developed corresponding theories and technologies, thus gradually forming an independent branch of acoustics.

[0036] This application mainly involves audio signal processing in the field of acoustics. Audio signals are signals that represent mechanical waves and are information carriers of the wavelength and intensity changes of mechanical waves. According to the characteristics of mechanical waves, they can be divided into regular signals and irregular signals. Among them, regular signals can be divided into music, etc. Regular signals are a continuously changing analog signal that can be represented by a continuous curve.

[0037] The digitization of signals is to convert continuously changing analog signals into discrete digital signals, which generally requires three steps: sampling, quantization, and encoding. Sampling refers to replacing the original continuous signal in time with a sequence of signal sample values ​​at regular intervals, that is, discretizing the analog signal in time; quantization is to use a finite number of approximate original continuously changing amplitude values ​​to convert the continuous amplitude of the analog signal into a finite number of discrete values ​​with certain intervals; encoding is to represent the quantized discrete values ​​with binary digits according to certain rules.

[0038] The audio signal processing involved in this application mainly refers to how to collect audio data, how to collect the short-term characteristics of audio data, and how to use the collected audio data to achieve the purpose of identifying content. In this application, the keystroke content is mainly identified through audio data. Keystroke refers to the action of tapping the keyboard. The keyboard involved in this application mainly refers to an instruction and data input device for operating computer equipment. It also refers to a group of function keys arranged by the system to operate a machine or device. For example, it includes both typewriters and hard keyboards on computers. In addition, it can also refer to soft keyboards on touch screens.

[0039] In addition, the keyboards involved in this application include both the Qwerty-arranged keyboards we commonly use and keyboards with other sequential arrangements, such as the Dvorak keyboard. The Qwerty keyboard, also known as the full keyboard, is a keyboard layout in which the first six letters of the first row are Q, W, E, R, T, and Y. The audio-based keystroke content recognition method used in this application has nothing to do with the letter arrangement of the keyboard. Even if the arrangement order of the keys on the keyboard is changed, as long as the layout of the keyboard is known, the audio-based keystroke content recognition method involved in this application can still be used. For ease of explanation, the Qwerty keyboard, which is more commonly used in daily life, is taken as an example below.

[0040] See also Figure 1 , Figure 1 : is a flowchart of a method for identifying keystroke content based on audio provided by the present application. The execution subject in the embodiment of the present application can be a computer device or a computer device cluster composed of multiple computer devices. The computer device can be a server or a terminal device. Therefore, the execution subject in the embodiment of the present application can be a server or a terminal device, or can be composed of a server and a terminal device, such as Figure 1 As shown, the method may include:

[0041] Step S101, receiving the audio of keystrokes;

[0042] The original audio of keystrokes is collected through a microphone. When collecting the audio of keystrokes, multiple microphones are needed to collect multiple audio data for the same keystroke sequence.

[0043] See also Figure 2 , Figure 2 It is a time domain diagram of the audio generated by a keystroke, where the horizontal axis is time and the vertical axis is loudness. According to the waveform generated by a keystroke, the keystroke can be divided into three parts. The first part is the touch segment, when the finger touches the keyboard, causing the keyboard button to vibrate slightly and produce sound; the second part is the press segment, when the finger presses the button and gradually presses to the bottom, a gradually weakening sound is generated; the third part is the release segment, when the finger releases the keyboard button, the button pops up and produces sound. Since the touch segment is usually connected to the press segment, it is regarded as a whole and is referred to as the key segment below.

[0044] It should be noted that for the soft keyboard on the touch screen, since the finger pressing on the screen and the finger lifting do not involve the button pressing and popping up behaviors in the hard keyboard, there is no release segment, but the key segment can still be located from the audio.

[0045] Furthermore, the sampling points and recording time can be converted through the sampling frequency, which is also called sampling speed or sampling rate. It defines the number of samples extracted from the continuous signal and composed into discrete signals per unit time. It is expressed in Hertz (Hz). The number of sampling points multiplied by the sampling frequency is the corresponding time. Taking the sampling frequency fs = 44100 Hz as an example, it means that the microphone samples 44100 signal samples per second. Assuming that the number of sampling points at the beginning of the key segment is n = 441, the time of the starting point is n*1 / fs = 0.01s.

[0046] Step S102, locating the keystroke segment in the audio;

[0047] Locating the keystroke segment in the audio mainly refers to finding the start and end sampling points (or time) of the keystroke segment and the release segment in the audio.

[0048] Optionally, the energy maximum in the audio waveform can be found through the short-time energy value, and then the key segment and the release segment can be located. The short-time energy value in the window can be calculated using a sliding window. The calculation formula of the short-time energy is:

[0049]

[0050] Where E i is the short-time energy value of the i-th window, x i (n) represents the audio signal of the i-th window, and n is the index of the sampling point in the window.

[0051] Find the maximum point of the short-time energy value of each window, set the threshold of the short-time energy, and when the short-time energy value of the window is greater than or equal to the threshold, it is considered that a keystroke has occurred, thereby locating the keystroke position of the audio waveform.

[0052] Furthermore, after the energy value maximum point is determined, the starting point of the key segment and the release segment is determined using a preset range. The preset range can be set according to the actual application scenario. For example, the starting point of the key segment can be set to 10ms before and 90ms after the sampling point where the energy value maximum is located.

[0053] Furthermore, audio features such as Mel-frequency cepstral coefficients (MFCC) can be extracted from the located audio segments. Mel-frequency cepstral coefficients (MFCC) are one of the common features in speech signal processing and are widely used in automatic speech and speaker recognition. The keystroke segments are classified using machine learning methods, thereby separating the keystroke segments from waveforms generated by other noises.

[0054] Specifically, an audio segment prediction network may be used to identify keystroke segments. The segment prediction network is a trained network that can classify audio segments and can separate keystroke segments from other audio segments.

[0055] Furthermore, when predicting audio segments, MFCC can be used as the audio feature of the training set. When classifying, the MFCC coefficients of the segment to be classified can be extracted, and the audio segment prediction network can be used to identify the keystroke segment and distinguish it from the non-keystroke audio segment. The audio segment is pre-emphasized, framed, and windowed, and then the windowed signal is discrete Fourier transformed, and then filtered using a Mel-scale filter bank. Because the frequency domain signal has a lot of redundancy, the filter bank can simplify the amplitude of the frequency domain. For the amplitude spectrum obtained by discrete Fourier transform, each filter is multiplied and accumulated by frequency, and the obtained value is the energy value of the frame data in the corresponding frequency band of the filter. Take the logarithm of the energy value obtained, and then perform a discrete cosine transform. The 2nd to 13th coefficients after the discrete cosine transform are the Mel frequency cepstrum coefficients MFCC.

[0056] For ease of understanding, after locating the key segments and release segments in the audio, these segments are indexed and numbered for subsequent processing. P can be used as a marker for key segments, and R can be used as a marker for release segments. For example, when typing "hello", five key segments and five release segments are generated, and these segments are captured by multiple microphones. The positions of these segments in the audio are located separately, such as the key segment of "h" recorded by microphone 1 is marked as P11, the release segment of "h" recorded by microphone 1 is marked as R11, the key segment of "h" recorded by microphone 2 is marked as P21, the release segment of "h" recorded by microphone 2 is marked as R21, and the key segment of "e" recorded by microphone 2 is marked as P22, thereby indexing all segments.

[0057] Step S103, determining one or more keystroke constraint division methods according to the keyboard arrangement rule;

[0058] Optionally, when performing keystroke constraint division, the keys on the keyboard can be divided into left and right parts according to the operation of the left and right hands when typing. Taking the Qwerty keyboard as an example, "T", "G", "V" and the keys to their left are the left half of the keyboard, and "Y", "H", "B" and the keys to their right are the right half of the keyboard. Please refer to Table 1 below, "Q", "W", "E", "R", "T", "A", "S", "D", "F", "G", "Z", "X", "C", and "V" are divided into the left side of the keyboard, and "Y", "U", "I", "O", "P", "H", "J", "K", "L", "B", "N", and "M" are divided into the right side of the keyboard.

[0059] Table 1

[0060] Left Right QWERTASDFGZXCV YUIOPHJKLBNM

[0061] Furthermore, in order to further distinguish the keys on the same side, the keys on the same side can be subdivided into three types, including the same key, the adjacent keys (adjacent keys), and the farther keys. The adjacent keys refer to the keys surrounding the key, while the farther keys are the keys on the same side except the adjacent keys and the same key. Please refer to Table 2 below. Taking the "J" key as an example, the adjacent keys include "Y", "U", "I", "H", "K", "N", and "M", the farther keys include "O", "P", "L", and "B", and the keys on the opposite side include all the keys on the left.

[0062] Table 2

[0063]

[0064]

[0065] Optionally, the entire keyboard can be treated as a whole, and the distances between keys can be divided into three types. The first type includes adjacent keys and identical keys (adjacent), the second type adds an outer circle of keys (closer) to the former type, and the remaining keys are the fourth type (farther). Please refer to Table 3 below. Taking the "F" key as an example, the identical key is "F", the adjacent keys are "F", "E", "R", "T", "D", "G", "C", "V", "B", the closer keys include "F", "E", "R", "T", "D", "G", "C", "V", "B", "W", "S", "X", "Y", "H", and the farther keys are "Q", "A", "Z", "U", "J", "N", "I", "K", "M", "O", "L", "P". Since the positional relationship between keys constitutes a set of constraints when predicting keystroke content, the method of dividing the relative positions of keys determines the effect of prediction.

[0066] Table 3

[0067]

[0068]

[0069] Furthermore, different keystroke relative position prediction methods can be used to estimate the relative position of the previously located keystroke audio segment, that is, the constraint position corresponding to each keystroke can be obtained based on the position relationship between the demarcated keys.

[0070] Step S104, generating a keystroke constraint set using a keystroke relative position prediction method, wherein the keystroke relative position prediction method includes a TDOA method and a correlation sorting method, and the keystroke relative position prediction method corresponds to the keystroke constraint division method in a one-to-one manner;

[0071] In the present invention, the relative position of keystrokes mainly refers to the relationship between the positions of each keystroke, and the relationship between these keystroke positions is called a constraint. Different relative position prediction methods can be selected to correspond to different keystroke constraint division methods to generate a keystroke constraint set.

[0072] Optionally, the TDOA method can be used for relative position prediction. TDOA positioning is a positioning method using time difference. By measuring the time it takes for the signal to arrive at the monitoring station, the distance of the signal source can be determined. Using the distance from the signal source to each monitoring station (using the monitoring station as the center and the distance as the radius to draw a circle), the position of the signal can be determined. In the present invention, this method is used as a method for estimating the relative position of keystrokes. Since the accuracy of the TDOA method will be lower when the number of microphones is small, in the present invention, it is mainly used to make a rough prediction of the area where the key is located, rather than as a method for identifying a single key.

[0073] Optionally, the offset time of the highest point of the key audio loudness is used as the TDOA value. First, the key interval obtained in step S102 is found, and the maximum value of the key interval audio loudness generated by each key press in multiple microphones is calculated as the TDOA value. The calculation of the TDOA method is as follows:

[0074]

[0075] Among them, i and j represent two microphones, A i (t i ) represents the microphone i at the keystroke time t i Audio clip of A j (t j ) represents the microphone j at the keystroke time t j The audio clip, where t i and t jRefers to different keystroke timing nodes, A i (t i ) and A j (t j ) means that t i and t j The audio clip that serves as the starting point of the keystroke event.

[0076] Optionally, the offset time of the key is calculated using the correlation coefficient, and the key segments generated by the same key action in each microphone are divided into the same group, and then the TDOA values ​​are calculated in pairs. During the calculation, one key segment area in the same group is kept stationary and used as a comparison segment, and then the other key segment is offset using a sliding window of the same segment size, and the correlation coefficient between the key segment area and the comparison segment after each offset is calculated to obtain a set of correlation coefficient values, where the offset value of the sliding window corresponding to the maximum correlation coefficient is the required TDOA value of the key in the two microphones.

[0077] The calculation formula of correlation coefficient cc is:

[0078]

[0079] Among them, A i (t) represents the audio segment of microphone i, t is the time of the keystroke segment in microphone i, n is the offset value, and A j (tn) represents the audio data after the keystroke segment window of microphone j is offset by n sampling points based on time t. i represents the average value of the audio clip from microphone i, μ j represents the average value of the audio segment for microphone j.

[0080] The final TDOA value can be expressed as:

[0081] t TDOA (n) = arg max cc(n)

[0082] By offsetting the number of sampling points, we can roughly get the difference between the position of the keystroke and the distance between the two microphones. The formula is:

[0083]

[0084] Where f is the sampling frequency of the microphone, c is the speed of sound, Δk is the distance difference between the button and the two microphones, and t TDOAis the offset of the sampling point, that is, the TDOA value obtained above. Using the property that the distance difference from a point on the hyperbola to the two foci is equal, a hyperbola can be drawn through the distance difference Δk obtained, and each microphone can be combined with other microphones, with a width similar to the size of the key, and different hyperbolas can be drawn to find the point with the most coverage.

[0085] Since the hyperbola covers more than one key, the positions can be predicted based on the keys with the most coverage. According to the principle of maximum coverage, the two farthest keys in the candidate area are selected to give position constraints. For example, the pre-selected area obtained by the TDOA method for keystroke segment 1 is "Y", "H", "B", "U", "J", and "N", and the pre-selected area obtained by the TDOA method for keystroke segment 2 is "I", "K", "M", "O", and "L". According to the principle of maximum coverage, the two farthest keys "B" and "O" in the candidate area are selected as the basis for dividing the key constraints.

[0086] Furthermore, a keystroke constraint division method can be selected to correspond to it. Taking the basis for dividing key constraints as "B" and "O" as an example, referring to the key constraint division method shown in Table 2, the key constraint of the unknown key corresponding to these two key fragments is "farther key"; if the key constraint division method shown in Table 3 is used, the key constraint of the unknown key corresponding to these two key fragments is "farther key".

[0087] Optionally, when performing relative position estimation, a correlation comparison method can be used to calculate the correlation coefficient between each keystroke segment and other keystroke segments. In this case, it is not limited to the keystroke segments collected by the same microphone, but also includes all keystroke segments collected by other microphones. The calculation of the correlation coefficient here can be in the time domain or in the frequency domain, and the calculation formula of the correlation coefficient is:

[0088]

[0089] Among them, A i (t p ) is the audio of the pth keystroke segment collected by microphone i, A j (t q ) is the audio of the qth keystroke segment collected by microphone j, μ i represents the average value of the audio clip in microphone i, μ j represents the average value of the audio clip from microphone j.

[0090] Furthermore, a threshold calculation method is selected according to the obtained correlation coefficient to sort the similarities between each key press and other keys in ascending order. pqThe index of the audio segment is marked. The first index p is the number of the microphone collected, and the second index q is the order of the collected audio segments. For example, assuming there are two microphones and three key segments are divided in the keystroke audio, the index at this time is A 11 , A 12 , A 13 , A 21 , A 22 , A 23 Among them A 11 , A 12 , A 13 are the three key fragments extracted by microphone 1, A 21 , A 22 , A 23 are the three key fragments extracted by microphone 2; from another perspective, A 11 , A 21 is the audio segment generated by the same keystroke in the two microphones. At this time, when calculating the correlation between the first key and other key audio, calculate A 11 With A 12 , A 13 , A 21 , A 22 , A 23 The correlation coefficient and A 21 With A 11 , A 12 , A 13 , A 22 , A 23 The correlation coefficient is denoted as cc(ip:jq), which is the correlation coefficient between the p-th audio segment of the ith microphone and the q-th audio segment of the j-th microphone. The similarity of the two key presses is calculated by the correlation coefficient, and the similarity reflects the degree of similarity between the two key presses.

[0091] Optionally, the similarity can be calculated by dividing the audio clips of the two key presses to be compared in different microphones into two groups, that is, when calculating the similarity between the a-th key press and the b-th key press, the audio clips of the a-th key press in each microphone are divided into group A, and the audio clips of the b-th key press in each microphone are divided into group B, that is, in the correlation coefficient combination cc(ip:jq) obtained above, p is set to a and q is set to b, and the maximum value thereof is taken as the similarity of the two key presses, that is:

[0092] sim=max(cc(1a:1b),cc(1a:2b),...,cc(1a:mb),cc(2a:1b),cc(2a:2b),...,cc(2a:mb),...,cc(ma:mb))

[0093] Here, m is the number of microphones, and the number of microphones m here can be two or more.

[0094] Optionally, the similarity can be calculated by taking the average of all values ​​when calculating the correlation coefficients of the audio segments in group A and group B in pairs as the similarity of the audios of the two key actions, that is:

[0095]

[0096] Here, m is the number of microphones, and the number of microphones m here can be two or more.

[0097] Sort the similarity values ​​between each key press and other key fragments, record the order of similarity between each key fragment and other key fragments, and sum them up, set the first threshold and the second threshold respectively, when the order sum is less than the first threshold, the key constraint of the unknown key corresponding to the two key fragments is "close", when the order sum is between the first threshold and the second threshold, the key constraint of the unknown key corresponding to the two key fragments is "closer", when the order sum is greater than the second threshold, the key constraint of the unknown key corresponding to the two key fragments is "farther".

[0098] Optionally, the first threshold can be set to 3, and the second threshold can be set to 5. Please refer to Table 4 below. Taking the second column of the fourth row as an example, assuming that the table represents the similarity order of two key fragments P1 and P2 and the corresponding key constraints, it can be seen that P1 is ranked first in the similarity ranking of P2, and P2 is ranked third in the similarity ranking of P1. Therefore, the key constraints of the unknown keys corresponding to the two key fragments correspond to "closer" in the second column of the fourth row.

[0099] Table 4

[0100] 1 2 3 4 ... N-1 N 1 similar similar Closer Closer 2 similar Closer Closer 3 Closer Closer 4 Closer ... N-1 Farther N Farther Farther

[0101] Step S105, selecting a constraint subset from the keystroke constraint set;

[0102] Since the constraints in the constraint set are not completely correct, there will be wrong constraints. If all constraints are selected to guess the keystroke content, there will often be contradictory constraint subsets. Take two keystrokes A and B as an example. In one method, the constraint of the two keystrokes is that A and B are close, and in another method, the constraint of the two keystrokes is that A and B are far away, so it is necessary to avoid contradictory constraints when selecting. At the same time, since even a single wrong constraint may cause a large error in the subsequent results, only some constraints need to be selected from the keystroke constraint set as the constraint subset for subsequent identification of the keystroke content.

[0103] Optionally, a proportion is set to select a portion from the non-contradictory constraints as the keystroke constraints used to finally restore the keystroke content, and the selection proportion may be 80%.

[0104] Step S106, restoring the keystroke content from the constraint subset in combination with the corpus.

[0105] Restore the keystrokes based on the constraint subset. A corpus refers to a large-scale electronic text library that has been scientifically sampled and processed, which contains language materials that have actually appeared in the actual use of the language. Convert the words and sentences that appear in the corpus into the same way as the constraints. Take the word "how" as an example. There are three letters in total. On the qwerty keyboard, the first key of the word is closer to the second key, the first key is farther from the third key, and the second key is farther from the third key. Match the constraint subset with the constraints of the words with the same number of letters in the corpus, and the keystrokes that meet the conditions are the alternatives.

[0106] Since the restoration of keystroke content mainly depends on the constraint subset, any kind of content, such as commonly used Chinese and English, can be restored through the constraints on the keystroke position.

[0107] Optionally, the corpus used can be web page domain names.

[0108] It is understandable that in the embodiment of the present invention, the corpus can also be subjected to constraint processing, that is, a corresponding constraint dictionary is formed according to the actual position relationship of each character / word on the keyboard. The constraint dictionary records the position relationship of each letter or symbol on the keyboard with the letters or symbols at other positions on the keyboard based on the constraint division method while retaining the original word.

[0109] See also Figure 3 , Figure 3 1 is a schematic diagram of the structure of an audio-based keystroke content recognition device provided by the present application. The recognition device may be a computer program (including program code) running in a computer device. For example, the audio-based keystroke content recognition device is an application software that can be used to execute the corresponding steps of the method provided in the embodiment of the present application. Figure 3 As shown, the audio-based keystroke content recognition device 1 may include:

[0110] An audio receiving module 11 , an audio processing module 12 , a constraint dividing module 13 , a constraint generating module 14 , a constraint selecting module 15 and a content identifying module 16 .

[0111] The audio receiving module 11 is used to receive the audio when typing.

[0112] The original audio of keystrokes is collected through a microphone. When collecting the audio of keystrokes, multiple microphones are needed to collect multiple audio data for the same keystroke sequence.

[0113] An audio processing module 12, used to locate keystroke segments in the audio;

[0114] Locating the keystroke segment in the audio mainly refers to finding the start and end sampling points (or time) of the keystroke segment and the release segment in the audio.

[0115] Optionally, the energy maximum in the audio waveform can be found through the short-time energy value, and then the key segment and the release segment can be located. The short-time energy value in the window can be calculated using a sliding window. The calculation formula of the short-time energy is:

[0116]

[0117] Where E i is the short-time energy value of the i-th window, x i (n) represents the audio signal of the i-th window, and n is the index of the sampling point in the window.

[0118] Find the maximum point of the short-time energy value of each window, set the threshold of the short-time energy, and when the short-time energy value of the window is greater than or equal to the threshold, it is considered that a keystroke has occurred, thereby locating the keystroke position of the audio waveform.

[0119] Furthermore, after the energy value maximum point is determined, the starting point of the key segment and the release segment is determined using a preset range. The preset range can be set according to the actual application scenario. For example, the starting point of the key segment can be set to 10ms before and 90ms after the sampling point where the energy value maximum is located.

[0120] Furthermore, audio features such as Mel-frequency cepstral coefficients (MFCC) can be extracted from the located audio segments. Mel-frequency cepstral coefficients (MFCC) are one of the common features in speech signal processing and are widely used in automatic speech and speaker recognition. The keystroke segments are classified using machine learning methods, thereby separating the keystroke segments from waveforms generated by other noises.

[0121] Specifically, an audio segment prediction network may be used to identify keystroke segments. The segment prediction network is a trained network that can classify audio segments and can separate keystroke segments from other audio segments.

[0122] Furthermore, when predicting audio segments, MFCC can be used as the audio feature of the training set. When classifying, the MFCC coefficients of the segment to be classified can be extracted, and the audio segment prediction network can be used to identify the keystroke segment and distinguish it from the non-keystroke audio segment. The audio segment is pre-emphasized, framed, and windowed, and then the windowed signal is discrete Fourier transformed, and then filtered using a Mel-scale filter bank. Because the frequency domain signal has a lot of redundancy, the filter bank can simplify the amplitude of the frequency domain. For the amplitude spectrum obtained by discrete Fourier transform, each filter is multiplied and accumulated by frequency, and the value obtained is the energy value of the frame data in the corresponding frequency band of the filter. Take the log of the energy value obtained, and then perform a discrete cosine transform. The 2nd to 13th coefficients after the discrete cosine transform are the Mel frequency cepstrum coefficients MFCC.

[0123] The constraint division module 13 is used to determine one or more keystroke constraint division methods according to the keyboard arrangement rules; the keystroke relative position in the present invention mainly refers to the relationship between each keystroke position, and the relationship between these keystroke positions is called a constraint. Different relative position prediction methods can be selected to correspond to different keystroke constraint division methods to generate a keystroke constraint set.

[0124] Optionally, when performing keystroke constraint division, the keys on the keyboard can be divided into left and right parts according to the operation of the left and right hands when typing. Taking the qwerty keyboard as an example, "T", "G", "V" and the keys to their left are the left half of the keyboard, and "Y", "H", "B" and the keys to their right are the right half of the keyboard. Referring to Table 1, "Q", "W", "E", "R", "T", "A", "S", "D", "F", "G", "Z", "X", "C", and "V" are divided into the left side of the keyboard, and "Y", "U", "I", "O", "P", "H", "J", "K", "L", "B", "N", and "M" are divided into the right side of the keyboard.

[0125] Furthermore, in order to further distinguish the keys on the same side, the keys on the same side can be subdivided into three types, including the same key, the adjacent keys (adjacent keys), and the farther keys. The adjacent keys refer to the keys surrounding the key, while the farther keys are the keys on the same side except the adjacent keys and the same key. Please refer to Table 2. Taking the "J" key as an example, the adjacent keys include "Y", "U", "I", "H", "K", "N", and "M", the farther keys include "O", "P", "L", and "B", and the keys on the opposite side include all the keys on the left.

[0126] Optionally, the entire keyboard can be treated as a whole, and the distances between keys can be divided into three types: the first type includes adjacent keys and the same keys (adjacent), the second type includes keys in a further circle (closer) based on the first type, and the remaining keys are the fourth type (farther). Since the positional relationship between keys constitutes a set of constraints when predicting keystrokes, the method of dividing the relative positions of keys determines the effect of prediction.

[0127] The constraint generation module 14 is used to select a plurality of keystroke relative position prediction methods to generate a keystroke constraint set.

[0128] Optionally, the TDOA method can be used for relative position prediction. TDOA positioning is a method for positioning using time difference. By measuring the time it takes for the signal to arrive at the monitoring station, the distance of the signal source can be determined. Using the distance from the signal source to each monitoring station (using the monitoring station as the center and the distance as the radius to draw a circle), the position of the signal can be determined. In the present invention, this method is used as a method for estimating the relative position of keystrokes. Since the accuracy of the TDOA method will be lower when the number of microphones is small, in the present invention, it is mainly performed on the area where the keys are located, rather than as a method for identifying individual keys.

[0129] Optionally, the offset time of the highest point of the key audio loudness is used as the TDOA value. First, the key interval obtained in step S102 is found, and the maximum value of the key interval audio loudness generated by each key press in multiple microphones is calculated as the TDOA value. The calculation of the TDOA method is as follows:

[0130]

[0131] Among them, i and j represent two microphones, A i (t i ) represents the microphone i at the keystroke time t i Audio clip of A j (t j ) represents the microphone j at the keystroke time t j The audio clip, where t i and t j Refers to different keystroke timing nodes, A i (t i ) and A j (t j ) means that t i and t j The audio clip that serves as the starting point of the keystroke event.

[0132] Optionally, the offset time of the key is calculated using the correlation coefficient, and the key segments generated by the same key action in each microphone are divided into the same group, and then the TDOA values ​​are calculated in pairs. During the calculation, one key segment area in the same group is kept stationary and used as a comparison segment, and then the other key segment is offset using a sliding window of the same segment size, and the correlation coefficient between the key segment area and the comparison segment after each offset is calculated to obtain a set of correlation coefficient values, where the offset value of the sliding window corresponding to the maximum correlation coefficient is the required TDOA value of the key in the two microphones.

[0133] The calculation formula of correlation coefficient cc is:

[0134]

[0135] Among them, A i (t) represents the audio segment of microphone i, t is the time of the keystroke segment in microphone i, n is the offset value, and A j (tn) represents the audio data after the keystroke segment window of microphone j is offset by n sampling points based on time t. i represents the average value of the audio clip from microphone i, μ j represents the average value of the audio segment for microphone j.

[0136] The final TDOA value can be expressed as:

[0137] t TDOA (n) = arg max cc(n)

[0138] By offsetting the number of sampling points, we can roughly get the difference between the position of the keystroke and the distance between the two microphones. The formula is:

[0139]

[0140] Where f is the sampling frequency of the microphone, c is the speed of sound, Δk is the distance difference between the button and the two microphones, and t TDOA is the offset of the sampling point, that is, the TDOA value obtained above. Using the property that the distance difference from a point on the hyperbola to the two foci is equal, a hyperbola can be drawn through the distance difference Δk obtained, and each microphone can be combined with other microphones, with a width similar to the size of the key, and different hyperbolas can be drawn to find the point with the most coverage.

[0141] Since the hyperbola will cover more than one key, the position can be predicted based on the keys with the most covered positions. According to the principle of maximum coverage, the two farthest keys in the candidate area are selected to give position constraints.

[0142] Furthermore, a keystroke constraint division method can be selected to correspond to it. Taking the basis for dividing key constraints as "B" and "O" as an example, if the key constraint division method shown in Table 2 is used, the key constraint of the unknown key corresponding to these two key fragments is "farther key"; if the key constraint division method shown in Table 3 is used, the key constraint of the unknown key corresponding to these two key fragments is "farther key".

[0143] Optionally, when performing relative position estimation, a correlation comparison method can be used to calculate the correlation coefficient between each keystroke segment and other keystroke segments. In this case, it is not limited to the keystroke segments collected by the same microphone, but also includes all keystroke segments collected by other microphones. The calculation of the correlation coefficient here can be in the time domain or in the frequency domain, and the calculation formula of the correlation coefficient is:

[0144]

[0145] Among them, A i (t p ) is the audio of the pth keystroke segment collected by microphone i, A j (t q ) is the audio of the qth keystroke segment collected by microphone j, μ i represents the average value of the audio clip in microphone i, μ j represents the average value of the audio segment in microphone j, where p and q refer to different keystroke segment numbers.

[0146] Furthermore, according to the obtained correlation coefficient, the threshold calculation method is selected for sorting, and the similarity between each key press and other keys is arranged in ascending order, starting with A pq The index of the audio segment is marked. The first index p is the number of the microphone collected, and the second index q is the order of the collected audio segments. For example, assuming there are two microphones and three key segments are divided in the keystroke audio, the index at this time is A 11 , A 12 , A 13 , A 21 , A 22 , A 23 Among them A 11 , A 12 , A 13 are the three key fragments extracted by microphone 1, A 21 , A 22 , A 23 are the three key fragments extracted by microphone 2; from another perspective, A 11 , A 21is the audio segment generated by the same keystroke in the two microphones. At this time, when calculating the correlation between the first key and other key audio, calculate A 11 With A 12 , A 13 , A 21 , A 22 , A 23 The correlation coefficient and A 21 With A 11 , A 12 , A 13 , A 22 , A 23 The correlation coefficient is denoted as cc(ip:jq), which is the correlation coefficient between the p-th audio segment of the ith microphone and the q-th audio segment of the j-th microphone. The similarity of the two key presses is calculated by the correlation coefficient, and the similarity reflects the degree of similarity between the two key presses.

[0147] Optionally, the similarity can be calculated by dividing the audio clips of the two key presses to be compared in different microphones into two groups, that is, when calculating the similarity between the a-th key press and the b-th key press, the audio clips of the a-th key press in each microphone are divided into group A, and the audio clips of the b-th key press in each microphone are divided into group B, that is, in the correlation coefficient combination cc(ip:jq) obtained above, p is set to a and q is set to b, and the maximum value thereof is taken as the similarity of the two key presses, that is:

[0148] sim=max(cc(1a:1b),cc(1a:2b),...,cc(1a:mb),cc(2a:1b),cc(2a:2b),...,cc(2a:mb),...,cc(ma:mb))

[0149] Here, m is the number of microphones, and the number of microphones m here can be two or more.

[0150] Optionally, the similarity can be calculated by taking the average of all values ​​when calculating the correlation coefficients of the audio segments in group A and group B in pairs as the similarity of the audios of the two key actions, that is:

[0151]

[0152] Here, m is the number of microphones, and the number of microphones m here can be two or more.

[0153] Sort the similarity values ​​between each keystroke and other keystroke fragments, record the order of similarity between each keystroke fragment and other keystroke fragments, and sum them up. Set the first threshold and the second threshold respectively. When the order sum is less than the first threshold, the key constraint of the unknown key corresponding to the two keystroke fragments is "close". When the order sum is between the first threshold and the second threshold, the key constraint of the unknown key corresponding to the two keystroke fragments is "close". When the order sum is greater than the second threshold, the key constraint of the unknown key corresponding to the two keystroke fragments is "far".

[0154] Optionally, the first threshold may be set to 3, and the second threshold may be set to 5.

[0155] A constraint selection module 15, configured to select a constraint subset from the keystroke constraint set

[0156] Since the constraints in the constraint set are not completely correct, there will be wrong constraints. If all constraints are selected to guess the keystroke content, there will often be contradictory constraint subsets. Take two keystrokes A and B as an example. In one method, the constraint of the two keystrokes is that A and B are close, and in another method, the constraint of the two keystrokes is that A and B are far away, so it is necessary to avoid contradictory constraints when selecting. At the same time, since even a single wrong constraint may cause a large error in the subsequent results, only some constraints need to be selected from the keystroke constraint set as the constraint subset for subsequent identification of the keystroke content.

[0157] Optionally, a proportion is set to select a portion from the non-contradictory constraints as the keystroke constraints used to finally restore the keystroke content, and the selection proportion may be 80%.

[0158] The content identification module 16 is used to restore the keystroke content from the constraint subset in combination with the corpus.

[0159] Restore the keystrokes based on the constraint subset. A corpus refers to a large-scale electronic text library that has been scientifically sampled and processed, which contains language materials that have actually appeared in the actual use of the language. Convert the words and sentences that appear in the corpus into the same way as the constraints. Take the word "how" as an example. There are three letters in total. On the qwerty keyboard, the first key of the word is closer to the second key, the first key is farther from the third key, and the second key is farther from the third key. Match the constraint subset with the constraints of the words with the same number of letters in the corpus, and the keystrokes that meet the conditions are the alternatives.

[0160] Since the restoration of keystroke content mainly depends on the constraint subset, any kind of content, such as commonly used Chinese and English, can be restored through the constraints on the keystroke position.

[0161] Optionally, the corpus used can be web page domain names.

[0162] It is understandable that in the embodiment of the present invention, the corpus can also be subjected to constraint processing, that is, a corresponding constraint dictionary is formed according to the actual position relationship of each character / word on the keyboard. The constraint dictionary records the position relationship of each letter or symbol on the keyboard with the letters or symbols at other positions on the keyboard based on the constraint division method while retaining the original word.

[0163] According to one embodiment of the present application, Figure 1 The steps involved in the audio-based keystroke content recognition method shown can be Figure 3 The various modules in the audio-based keystroke content recognition device 1 are executed. For example, Figure 1 The step S101 shown in FIG. 1 can be performed by Figure 3 The audio receiving module 11 is used to execute, Figure 1 The step S102 shown in FIG. 1 can be performed by Figure 3 The audio processing module 12 is used to perform; Figure 1 The step S103 shown in FIG. 1 can be performed by Figure 3 The constraint partitioning module 13 in the implementation is used to perform the Figure 1 The step S104 shown in FIG. 1 can be performed by Figure 3 The constraint generation module 14 is executed in Figure 1 The step S105 shown in FIG. 1 can be performed by Figure 3 The constraint selection module 15 is executed in, Figure 1 The step S106 shown in FIG. 1 can be performed by Figure 3 The content identification module 16 is used to perform the operation.

[0164] The present application provides a computer program product or a computer program, which includes a computer instruction, which is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the description of the above-mentioned image data processing method in the corresponding embodiment above, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0165] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0166] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0167] The method and related device provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application, and each flow and / or box of the method flow chart and / or structural diagram, and the combination of the flow chart and / or box in the flow chart and / or box diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing a flow chart or multiple flows and / or a box or multiple boxes in the structural diagram. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements a flow chart or multiple flows and / or a box or multiple boxes in the structural diagram. Function specified. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in a flowchart and / or one or more boxes in a structure diagram.

[0168] In the description of the present invention, it is necessary to understand that the terms "coaxial", "bottom", "one end", "top", "middle", "the other end", "upper", "one side", "top", "inside", "outside", "front", "center", "both ends" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0169] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0170] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for identifying keystroke content based on audio, characterized in that: include: Receiving audio when a key is struck; the audio is received via a plurality of microphones; Locate keystroke fragments in audio; Determine one or more keystroke constraint division methods according to keyboard arrangement rules; A keystroke constraint set is generated for the located keystroke fragment using a keystroke relative position prediction method corresponding to one or more keystroke constraint partitioning methods, wherein the keystroke relative position prediction method includes a TDOA method and a relevance ranking method; the relevance ranking method includes: Calculate the correlation coefficient between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone; calculating the similarity between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone according to the correlation coefficient between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone, wherein the similarity refers to the average or maximum value of the correlation coefficients between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone; The similarity between the keystroke fragment of the keystroke and the keystroke fragments of other keystrokes is sorted in descending order, that is, the one with the highest similarity is ranked first; Sum the similarity ranking values ​​of the keystroke segment of the keystroke and the keystroke segments of other keystrokes, and set a first threshold and a second threshold corresponding to the keystroke constraint division method; Sort and filter the similarities according to the first threshold and the second threshold to generate a keystroke constraint set; selecting a constraint subset from the keystroke constraint set; The keystroke content is restored from the constraint subset in combination with the corpus.

2. The method for identifying keystroke content based on audio according to claim 1, characterized in that: The similarity is sorted and filtered according to the first threshold and the second threshold to generate a keystroke constraint set, which includes adding the similarity sorts between two keystrokes. When the similarity sort sum is less than the first threshold, the positions of the two keys are close, indicating that the distance between the two keys is within one key at this time; when the similarity sort sum is greater than the first threshold and less than the second threshold, the positions of the two keys are closer, indicating that the distance between the two keys is within two keys at this time; when the similarity sort sum is greater than the second threshold, the positions of the two keys are farther, indicating that the distance between the two keys is outside the two keys at this time, and the position relationship between the two keys is used to generate a keystroke constraint set.

3. The method for identifying keystroke content based on audio according to claim 1, characterized in that: The TDOA method includes: Calculate the time difference of the same keystroke fragment received by each microphone for each keystroke; Determine the coverage of the keystroke by using the time difference; Divide the keystroke constraint according to the two farthest keys in the coverage of the two keystrokes; When the distance between two keys is within one key, the keystroke constraint is close; when the distance between two keys is within two keys, the keystroke constraint is closer; when the distance between two keys is outside two keys, the keystroke constraint is farther, and the positional relationship between the two keys is generated into a keystroke constraint set.

4. The method for identifying keystroke content based on audio according to claim 3, characterized in that: Determining the coverage of the key press by the time difference includes: Calculate the correlation coefficient of the same keystroke fragment received by each microphone for the current keystroke; Using windows of the same size to shift the keystroke fragments of the current keystroke, and respectively calculating the correlation coefficients corresponding to the same keystroke fragments received by each microphone during each sliding window; Select the offset point with the largest correlation coefficient, which is the offset sampling point number of the keystroke; The distance difference between the current keystroke position and the two microphones is obtained through the offset sampling points; A hyperbola is drawn through the distance difference to form a coverage area.

5. A keystroke content recognition device based on audio, characterized in that: include: An audio receiving module, used for receiving audio when keystrokes are made; the audio is received via a plurality of microphones; An audio processing module for locating keystroke segments in audio; A constraint division module, used to determine one or more keystroke constraint division methods according to keyboard arrangement rules; A constraint generation module, configured to generate a keystroke constraint set for the located keystroke fragments using a keystroke relative position prediction method corresponding to one or more keystroke constraint division methods, wherein the keystroke relative position prediction method includes a TDOA method and a relevance ranking method; The correlation sorting method adopted in the constraint generation module includes: Calculate the correlation coefficient between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone; calculating the similarity between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone according to the correlation coefficient between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone, wherein the similarity refers to the average or maximum value of the correlation coefficients between the keystroke segment of each keystroke and the keystroke segments of other keystrokes in each microphone; The similarity between the keystroke fragment of the keystroke and the keystroke fragments of other keystrokes is sorted in descending order, that is, the one with the highest similarity is ranked first; Sum the similarity ranking values ​​of the keystroke segment of the keystroke and the keystroke segments of other keystrokes, and set a first threshold and a second threshold corresponding to the keystroke constraint division method; Sort and filter the similarities according to the first threshold and the second threshold to generate a keystroke constraint set; A constraint selection module, configured to select a constraint subset from the keystroke constraint set; The content identification module is used to restore the keystroke content from the constraint subset in combination with the corpus.

6. The audio-based keystroke content recognition device according to claim 5, characterized in that: The TDOA method includes: Calculate the time difference of the same keystroke fragment received by each microphone for each keystroke; Determine the coverage of the keystroke by using the time difference; Divide the keystroke constraint according to the two farthest keys in the coverage of the two keystrokes; When the distance between two keys is within one key, the keystroke constraint is close; when the distance between two keys is within two keys, the keystroke constraint is closer; when the distance between two keys is outside two keys, the keystroke constraint is farther, and the positional relationship between the two keys is generated into a keystroke constraint set.

7. A computer device, characterized in that: It comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the audio-based keystroke content recognition method as described in any one of claims 1 to 4.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the audio-based keystroke content recognition method as described in any one of claims 1-4 are implemented.

Images