Handwriting recognition method and system based on ultrasonic waves, mobile terminal and server

Abstract

The application provides a handwriting recognition method and system based on ultrasonic waves, a mobile terminal and a server, the application combines ultrasonic waves with handwriting recognition, proposes a new input method, and through machine learning technology combining CTC loss and LSTM, realizes real-time recognition of handwriting input; the ultrasonic waves after being carried by the autocorrelation sequence improve the robustness to environmental changes, and in a cross-environment experiment, a high letter recognition accuracy is achieved; the machine learning model has good segmentation ability for the interval between words, can support continuous handwriting input recognition, solves the problem that long sequence input is difficult to segment in actual application, and partially solves the cross-environment problem faced by passive sensing, and shows great potential of ultrasonic wave passive recognition in commercial fields.

Description

Technical Field

[0001] This application relates to the field of handwriting recognition technology, and in particular to a handwriting recognition method, system, mobile terminal, and server based on ultrasound. Background Technology

[0002] Passive sensing systems utilize wireless signals generated by widely distributed wireless sensors in daily life to perceive human behavior. Currently, commercially available speakers and microphones in smart electronic products have the ability to transmit and receive ultrasonic waves. Utilizing these ultrasonic waves to build passive sensing systems can expand the capabilities of human-computer interaction. Compared to computer vision-based sensing methods, passive sensing poses less risk to user privacy and has significant energy efficiency advantages, making it of significant application value. Handwriting recognition based on ultrasound aligns better with human daily habits than keyboard or touchscreen input methods, and is also an important application direction for ultrasonic-based passive recognition systems.

[0003] However, current passive sensing systems based on ultrasound generally use gestures or lip movements for recognition. Gesture recognition's granularity is insufficient for handwriting recognition, while lip movements do not consider changes in spatial relationships. Current ultrasound motion recognition primarily focuses on single-command inputs, while handwriting recognition, as a continuous input method, has a longer sequence of activities and contains richer information.

[0004] In terms of technical solutions, systems focusing on single-command input first segment the collected sample data using a threshold-based algorithm, then train a system with recognition capabilities using machine learning methods. In practical use, users are required to actively pause after the action, allowing the algorithm to segment the data suitable for recognition. However, threshold-based segmentation algorithms are difficult to apply to continuous input, limiting user interaction in real-world applications. Furthermore, the machine learning algorithms in these systems often focus on classifying different individual actions, rarely considering the causal relationships within a sequence of activities, thus lacking the capability to be applied to handwriting recognition input systems. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a handwriting recognition method, system, mobile terminal and server based on ultrasound, so as to solve the problems of long sequence segmentation and cross-environment problems faced by passive sensing in the prior art.

[0006] To achieve the above and other related objectives, this application provides an ultrasonic-based handwriting recognition method applied to a mobile device equipped with a speaker and a microphone. The method includes: playing an ultrasonic audio signal loaded with a GTS original sequence modulated to the ultrasonic frequency band through the speaker; receiving an ultrasonic audio file reflected by handwriting actions near the mobile device through the microphone; and uploading the ultrasonic audio file to a server for receiving word results returned by the server.

[0007] In one embodiment of this application, the method for modulating the original GTS sequence into an ultrasonic frequency band includes: determining the target bandwidth and target frequency band to be modulated from the original GTS sequence; obtaining the frequency distribution of the original GTS sequence by Fourier transform based on the target bandwidth and the audio file frequency; extending the length of the original GTS sequence by adding zeros, and obtaining a signal with the target bandwidth by inverse Fourier transform; upsampling the signal to the target frequency band range by calculating with a sine wave of a specific frequency; wherein the characteristic frequency is the middle frequency of the target frequency band.

[0008] In one embodiment of this application, the method for determining the target frequency band is as follows: obtaining the upper limit of the audio sampling rate of the mobile terminal; obtaining the upper limit of the signal frequency of the mobile terminal according to the Nyquist sampling theorem; and determining the target frequency band based on the upper limit of the signal frequency and the lower limit of the ultrasonic frequency.

[0009] To achieve the above and other related objectives, this application provides an ultrasonic-based handwriting recognition method applied to a server. The method includes: acquiring an ultrasonic audio file representing handwriting actions uploaded by a mobile terminal; acquiring the GTS handwriting sequence contained in the ultrasonic audio file, and calculating the channel impulse response features representing the handwriting actions; extracting the real and imaginary parts of the channel impulse response features and the difference data of the corresponding time dimension, and inputting them as multi-dimensional features into a neural network model to extract deep features associated with different words or letters in the handwriting action sequence; and training a time-series model based on bidirectional LSTM and CTC loss function to generate probability prediction vectors for the word or letter dimensions corresponding to the handwriting action sequence, so as to identify the word corresponding to the handwriting action and transmit it back to the mobile terminal.

[0010] In one embodiment of this application, obtaining the GTS handwritten sequence contained in the ultrasonic audio file includes: downsampling the ultrasonic audio file and demodulating it to obtain a continuous GTS handwritten sequence; and segmenting the continuous GTS handwritten sequence into multiple GTS handwritten sequence segments.

[0011] In one embodiment of this application, the step of segmenting a continuous GTS handwriting sequence into multiple individual GTS handwriting sequences includes: performing window sliding on the continuous GTS handwriting sequences using the autocorrelation of the GTS sequences; calculating the Pearson correlation coefficient with the original GTS sequences respectively, and using positions with coefficients close to 1 or -1 as cutting points to segment multiple GTS handwriting sequence segments for extracting channel impulse response features respectively; wherein, the original GTS sequence is a GTS handwriting sequence containing characterizing handwriting actions that is pre-modulated to the ultrasonic frequency band by the mobile terminal to generate an ultrasonic audio signal, and is played back to collect the reflected signal.

[0012] In one embodiment of this application, the calculation of the channel impulse response characteristics characterizing the handwriting action includes: based on the LTI system, convolving the GTS handwritten sequence fragment r(t) with the original GTS sequence x(t) to obtain: r(t) = h(t) * x(t), where h(t) represents the channel impulse response signal; assuming there are L propagation paths under multipath effects, and the propagation delay of each path is τ. i The signal strength parameter for each path is a. i Then r(t) can be considered as the superposition of L reflected signals, which yields: Since the signal transmitted by the loudspeaker is discrete, h(t) is represented in matrix form, i.e.: h(t) = [h0, h1, ..., h L ] T , where h i This is called the i-th channel tap; x(t) is divided into P reference bits and L guard bits. At time i, the GTS handwritten sequence fragment is the result of multipath transformation from the original GTS sequence i to L+i bits: r(i) = [x L+i ,x L+i-1 ,...,x i ]*h(t)+n; where n is the noise constant; after transformation, we get: r(t)=Mx+N; M is the transformation matrix; at this point, the calculation of h(t) is transformed into an optimal estimation problem, to find the value that makes h(t) equal to the noise constant. The optimal value with the smallest difference from r estimate: By the least squares estimation method, we can obtain: h(t) = (M H M) -1 M H r(t); where the superscript H indicates the conjugate operator.

[0013] In one embodiment of this application, the time-series model constructed based on bidirectional LSTM and CTC loss function, used to train probability prediction vectors for handwritten action sequences corresponding to words or letters, includes: the LSTM reducing the amount of output data through a fully connected layer, and obtaining the probability of the letter corresponding to each unit through the SoftMax normalization function; given a word 'a', all outputs 'π' equivalent to 'a' are calculated as follows: Among them, B -1 (a) is the inverse function. The positive function B(π) = a means that π can be equivalent to the word a, and the inverse function means all such combinations of π. Pr(a|x) means the probability that the result is equivalent to a given output x.

[0014] To achieve the above and other related objectives, this application provides a mobile terminal, the mobile terminal comprising: a speaker for playing an ultrasonic audio signal loaded with a GTS original sequence modulated to the ultrasonic frequency band; a microphone for receiving an ultrasonic audio file reflected by handwriting actions near the mobile terminal; and a communicator for uploading the ultrasonic audio file to a server or receiving word results returned by the server.

[0015] To achieve the above and other related objectives, this application provides a server, which includes: a memory, a processor, and a communicator; the memory is used to store computer instructions; the processor executes the computer instructions to implement the method described above; the communicator is used to communicate with the mobile terminal described above to receive ultrasonic audio files or send word results corresponding to recognized handwriting actions.

[0016] To achieve the above and other related objectives, this application provides an ultrasonic-based handwriting recognition system, the system comprising: a mobile terminal as described above, used to acquire an ultrasonic audio file reflected by a corresponding handwriting action; and a server as described above, communicatively connected to the mobile terminal, used to identify the word result corresponding to the handwriting action based on the ultrasonic audio file, and transmit the word result back to the mobile terminal.

[0017] In summary, this application provides a handwriting recognition method, system, mobile terminal, and server based on ultrasound.

[0018] It has the following beneficial effects:

[0019] This application combines ultrasound with handwriting recognition to propose a novel input method with excellent word segmentation capabilities, supporting continuous handwriting input recognition. It solves the problem of segmenting long input sequences in practical applications; improves robustness to environmental changes; partially addresses the cross-environmental challenges faced by passive sensing; and demonstrates the enormous potential of passive ultrasound recognition in commercial applications. Attached Figure Description

[0020] Figure 1 The diagram shown is a scenario illustration of an ultrasonic-based handwriting recognition system according to an embodiment of this application.

[0021] Figure 2 The diagram shows a flowchart of an ultrasonic-based handwriting recognition method applied to a mobile device in one embodiment of this application.

[0022] Figure 3 The diagram shown is a flowchart illustrating an ultrasonic-based handwriting recognition method applied to a server in one embodiment of this application.

[0023] Figure 4 The diagram shown is a flowchart of an ultrasonic-based handwriting recognition system according to one embodiment of this application.

[0024] Figure 5 The diagram shown is a schematic representation of a CNN network model in one embodiment of this application.

[0025] Figure 6 The diagram shown is a schematic representation of a timing modeler in one embodiment of this application.

[0026] Figure 7 The diagram shown is a structural schematic of a server according to one embodiment of this application. Detailed Implementation

[0027] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0028] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Although the illustrations only show the components related to this application and are not drawn according to the actual number, shape and size of the components, the form, quantity and proportion of each component can be arbitrarily changed in actual implementation, and the layout of the components may also be more complex.

[0029] As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are to be interpreted inclusively, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition occur only when combinations of elements, functions, steps, or operations are inherently mutually exclusive in some way.

[0030] To address the aforementioned issues, this application provides an ultrasonic-based handwriting recognition method, system, mobile terminal, and server. This application aims to utilize ultrasonic signals from commercial acoustic sensors to establish a handwriting recognition input system suitable for mobile electronic devices. The main technical problems addressed include improving the resolution of ultrasonic signals, introducing the ability to recognize temporal sequence contexts, thereby improving the accuracy of handwritten text recognition, and removing the dependence of handwriting sequence action recognition on segmentation algorithms. Ultimately, this results in a mobile terminal system with real-time recognition capabilities for continuous handwriting input without restricting user input actions.

[0031] like Figure 1 The figure shows a schematic diagram of a scenario for an ultrasonic-based handwriting recognition system according to an embodiment of this application. As shown, the system includes a mobile terminal 100 and a server 200.

[0032] Specifically, the mobile terminal 100 includes:

[0033] Speaker 110 is used to play an ultrasonic audio signal loaded with a GTS original sequence modulated to the ultrasonic frequency band.

[0034] Microphone 120 is used to receive ultrasonic audio files reflected from handwriting gestures near the mobile device;

[0035] The communicator 130 is used to upload the ultrasonic audio file to the server or receive word results returned by the server.

[0036] For example, the mobile terminal 100 can be a smartphone, tablet, smartwatch, laptop, or other mobile device with a speaker 110, a microphone 120, and a communicator 130.

[0037] The server 200 is mainly used to receive ultrasonic audio files, perform a series of processing, identify the word result corresponding to the handwritten action, and then transmit the word result back to the mobile terminal 100.

[0038] In one or more feasible application scenarios, specific embodiments of the ultrasonic-based handwriting recognition system described in this application may be as follows:

[0039] For example, in practical applications, the handwriting recognition software used by users is implemented as an input method on mobile devices such as Android phones or Apple phones. The input method source code used includes the input method source code from AOSP (Android OpenSource Project), which has been customized for the application scenario and does not display the keyboard or other interfaces.

[0040] In terms of functionality, the system first plays a pre-generated ultrasonic audio signal through the speaker 110 in the direction of the handwriting action to reduce the amount of computation. Then, the reflected ultrasonic audio signal is received through the microphone 120 and converted into an audio file, which is then uploaded to the server 200 via the network.

[0041] Server 200 receives the audio file and completes the recognition of handwriting. It then sends the recognized words back to mobile device 100. The input method program on mobile device 100 receives the results via wireless network and inputs them into the corresponding input box on mobile device 100.

[0042] In order to accurately identify the word results corresponding to handwritten actions, the server 200 is also responsible for downsampling audio files, calculating channel impulse response characteristics and machine learning model results for recognition.

[0043] like Figure 2 The figure shows a flowchart illustrating an ultrasonic-based handwriting recognition method for mobile devices according to an embodiment of this application. As shown, the method includes:

[0044] Step S201: Play an ultrasonic audio signal loaded with the original GTS sequence modulated to the ultrasonic frequency band through a speaker.

[0045] It should be noted that this application introduces channel estimation technology from the field of mobile communications into the field of acoustics, embedding a highly autocorrelated GTS (Global System for Mobile Communications) sequence on the ultrasonic frequency band of a smartphone. GTS is an abbreviation for GSM (Global System for Mobile Communications) Training Sequence. Because the GTS sequence has good autocorrelation and can be processed using channel impulse response (CIR) feature extraction algorithms, this application uses the GTS sequence to characterize or reflect handwriting action features. Table 1 below shows the original GTS sequence.

[0046] Table 1. Original GTS sequences

[0047] 0 0 1 0 0 1 0 1 1 1 0 0 0 0 1 0 0 0 1 0 0 1 0 1 1 1

[0048] The table above shows a 26-bit GTS sequence. Its good autocorrelation can be understood as follows: for example, if two GTS sequences are concatenated to form a 52-bit vector, then 26 bits are progressively truncated and multiplied by the original GTS: x[0∶26](original GTS)*x[i∶i+26]; where i is the bit. The sequence with the best autocorrelation has only x[0∶26]*x[0∶26] being 11 (this GTS has 11 ones), while the others x[0∶26]*x[i∶i+26] are close to zero. This means that a sequence must be perfectly aligned with the GTS for the correlation coefficient to be 1; even a slight deviation will cause the correlation coefficient to drop sharply. The autocorrelation of GTS sequences has applications on the server side.

[0049] The method for modulating the original GTS sequence into an ultrasonic frequency band includes:

[0050] A. Determine the target bandwidth and target frequency band for modulation of the original GTS sequence;

[0051] The method for determining the target frequency band is as follows:

[0052] 1) Obtain the upper limit of the audio sampling rate on the mobile device;

[0053] 2) Obtain the upper limit of the signal frequency of the mobile terminal according to the Nyquist sampling theorem;

[0054] 3) Determine the target frequency band based on the upper limit of the signal frequency and the lower limit of the ultrasonic frequency.

[0055] GTS (Glass Transmission Signal) exhibits good autocorrelation, but since the frequency at which the GTS operates differs from the frequency of the audio signal, signal modulation is required to transmit ultrasound on a mobile device. Firstly, the current maximum audio sampling rate for smartphones is 48000Hz, which, according to the Nyquist sampling theorem, limits the supported signal frequency to 24kHz. Given that the lower limit of ultrasound frequency is 18kHz, the 26-bit GTS sequence needs to be modulated to a frequency range of 18kHz to 24kHz, thus defining the target frequency band.

[0056] B. Based on the target bandwidth and audio file frequency, the frequency distribution of the original GTS sequence is obtained through Fourier transform;

[0057] C. The length of the original GTS sequence is extended by adding zeros, and the signal with the target bandwidth is obtained by inverse Fourier transform;

[0058] D. The signal is upsampled to the target frequency band range by calculating a sine wave of a specific frequency; wherein the characteristic frequency is the middle frequency of the target frequency band.

[0059] In simple terms, this application uses a zero-addition algorithm to adjust the bandwidth and frequency distribution of the GTS. The target modulation bandwidth is B, and the audio file frequency is f. s The frequency distribution of the GTS is obtained through Fourier transform, and then the sequence length is extended using the zero-addition algorithm. The signal with bandwidth B is then obtained through inverse Fourier transform, and the signal sequence at this point also exhibits good autocorrelation characteristics. An upsampling algorithm calculated using a sine wave of a specific frequency can directly upsample the signal to the target frequency band. This specific frequency is the midpoint of the target frequency band; in the case of 18kHz to 24kHz, 21kHz is used as the midpoint.

[0060] It should be noted that the step of loading the ultrasonic audio signal of the original GTS sequence modulated to the ultrasonic frequency band can be processed outside the mobile device. For example, it can be processed by other external devices such as computers or servers, and then the ultrasonic audio signal loaded with the original GTS sequence modulated to the ultrasonic frequency band can be directly transferred to the mobile device. Of course, software capable of performing the above operations can also be installed on the mobile device so that the ultrasonic audio signal can be generated directly on the mobile device.

[0061] Step S202: Receive the ultrasonic audio file reflected by handwriting actions near the mobile device via the microphone.

[0062] In simple terms, ultrasonic audio radiates outwards. When handwriting is performed within a certain distance (e.g., within 30cm), the reflected sound is sent back by the hand. This reflected ultrasonic audio signal is then continuously received by a microphone. This signal contains characteristic information representing the handwriting action and can be packaged into an audio file. Subsequently, the GTS handwriting sequence can be demodulated via a server or mobile device to calculate the channel impulse response characteristics, which can then be used as input to a neural network.

[0063] Step S203: Upload the ultrasonic audio file to the server so as to receive the word results returned by the server.

[0064] The reflected ultrasonic audio signal is packaged into an audio file, which facilitates network transmission so that the server can identify and process the audio file to receive the word results returned by the server.

[0065] For example, the mobile terminal 100 can be a smartphone, tablet, smartwatch, laptop, or other mobile device with a speaker 110, a microphone 120, and a communicator 130.

[0066] like Figure 3 The figure shows a flowchart illustrating an ultrasonic-based handwriting recognition method applied to a server according to an embodiment of this application. As shown, the method includes:

[0067] Step S301: Obtain the ultrasonic audio file representing handwriting actions uploaded by the mobile device.

[0068] Simply put, this step corresponds to Figure 2 The method first plays an ultrasonic audio signal loaded with a GTS original sequence modulated to the ultrasonic frequency band through a speaker; then it receives an ultrasonic audio file reflected by handwriting actions near the mobile device through a microphone, and then uploads the ultrasonic audio file to the server, thereby the server obtains the ultrasonic audio file representing handwriting actions uploaded by the mobile device.

[0069] For a better understanding of the ultrasonic-based handwriting recognition method applied to servers, please refer to [reference needed]. Figure 4 The diagram shows a flowchart of an ultrasonic-based handwriting recognition system.

[0070] Step S302: Obtain the GTS handwriting sequence contained in the ultrasonic audio file, and calculate the channel impulse response characteristics that characterize the handwriting action.

[0071] In one embodiment of this application, obtaining the GTS handwritten sequence contained therein from the ultrasonic audio file includes:

[0072] A. The ultrasonic audio file is downsampled and demodulated to obtain a continuous GTS handwritten sequence;

[0073] B. Divide the continuous GTS handwritten sequence into multiple GTS handwritten sequence segments.

[0074] In some examples, the environmental information contained in the demodulated GTS handwritten sequences is quite complex and difficult to use directly for model training. In order to obtain high-level features that can reflect hand movements, this invention uses the CIR (channel impulse response) estimation algorithm to extract high-level features from ultrasound.

[0075] CIR calculations are performed on a single autocorrelation sequence, while the transmitted ultrasound is a continuous GTS sequence, and the start and end positions of each GTS segment are unknown when the signal is received.

[0076] To achieve this goal, this application requires dividing a continuous GTS handwritten sequence into multiple individual GTS handwritten sequences, specifically including:

[0077] 1) Utilize the autocorrelation of GTS sequences to perform window sliding on continuous GTS handwritten sequences;

[0078] 2) Calculate the Pearson correlation coefficient for each sequence segment within the current window, and use the positions with coefficients close to 1 or -1 as cutting points to segment multiple GTS handwritten sequence segments for extracting channel impulse response features. The original GTS sequence is generated by the mobile terminal pre-modulating the ultrasonic frequency band to generate an ultrasonic audio signal, and then playing it to collect the reflected GTS handwritten sequence containing characteristics of handwriting.

[0079] First, the transmitted continuous GTS handwritten sequence needs to be segmented into multiple individual GTS handwritten sequences. At this point, the autocorrelation of the GTS sequences is utilized to perform sliding window segmentation on the received GTS handwritten sequences. Then, the Pearson correlation coefficient is calculated between each sequence and the original transmitted GTS sequence. The peak value of the coefficient is close to 1 or -1, indicating the best correlation with the transmitted sequence. Therefore, the peak value is used as the cutting point to segment the continuous GTS handwritten sequence into segments for channel impulse response (CIR) feature extraction. It should be noted that the purpose of segmentation is twofold: firstly, to adjust the GTS sequence length to segment the input or sample data; and secondly, to identify the starting point of each sequence within the continuous GTS handwritten sequence.

[0080] Step S303: Extract the real part, imaginary part, and the difference data of the corresponding time dimension of the channel impulse response features, and input them as multi-dimensional features into the neural network model to extract deep features associated with different words or letters in the handwritten action sequence.

[0081] In one embodiment of this application, the real part, imaginary part, and corresponding time dimension difference data of the extracted channel impulse response features are mainly for the segmented GTS handwritten sequence fragments.

[0082] Specifically, after segmentation, the propagated signal sequence r(t) is obtained. CIR estimation techniques are based on modeling the entire ultrasonic wave propagation process as a linear time-invariant (LTI) system. Let the transmitted signal sequence be x(t). The LTI model considers the received signal as the result of convolutional transformation of the transmitted signal, i.e.:

[0083] r(t) = h(t) * x(t);

[0084] Where h(t) represents the channel impulse response signal.

[0085] Under the multipath effect, assume there are L propagation paths, and the propagation delay of each path is τ. i The signal strength parameter for each path is a. i Then r(t) can be considered as the superposition of L reflected signals, which yields:

[0086]

[0087] The main purpose here is to calculate h(t), also known as the channel impulse response (CIR). Since the signal transmitted by the loudspeaker is discrete, h(t) is represented in matrix form, i.e.:

[0088] h(t) = [h0, h1, ..., h L ] T ;

[0089] Among them, h i It is called the i-th channel tap.

[0090] After segmenting the GTS, assuming the original GTS sequence is x and the received GTS handwritten sequence fragment data is r(t), the CIR estimation model divides x(t) into P reference bits and L guard bits to ensure the accuracy of the estimation results. At time i, the received GTS handwritten sequence is the result of multipath transformation from the original GTS sequence i to L+i bits, i.e.:

[0091] r(i) = [x L+i ,x L+i-1 ,...,x i ]*h(t)+n;

[0092] Where n is the noise constant.

[0093] Therefore, the received frame data can be represented as:

[0094] r(t) = Mx + N;

[0095]

[0096] Where M is the transformation matrix.

[0097] Thus, the calculation of the channel impulse response (CIR) characteristic is transformed into an optimal estimation problem, finding the value that makes the channel impulse response characteristic (CIR) such that... The optimal value with the smallest difference from r estimate:

[0098]

[0099] We can obtain the following from the least squares estimation method:

[0100] h(t)=(M H M) -1 M H r(t);

[0101] Here, the superscript H denotes the conjugate operator. Thus, this invention calculates a CIR of length P from an i-bit signal sequence.

[0102] In this application, the calculated CIR features are time-distance distributed data directly related to user actions. However, the similarity between handwritten actions is greater than the difference between different words. Therefore, in order to extract deep features related to different words and letters, this application uses a CNN network to extract local features of CIR.

[0103] CIR data includes action-related data and environment-related static data. To enable the classifier to better learn handwritten action-related features, this application processes CIR into differential CIR (dCIR) during the preprocessing stage. That is, given the time-channel two-dimensional distribution of CIR data h(t,n), dCIR is the data of the next frame minus the previous frame in the time dimension.

[0104] d(t,n)=h(t,n)-h(t-1,n),t>0;

[0105] In addition, CIR is complex data, so the real part, imaginary part and difference data are concatenated together as multidimensional features and then input into the neural network.

[0106] Step S304: A time-series model based on bidirectional LSTM and CTC loss function is used to train probability prediction vectors for words or letters corresponding to handwritten action sequences, so as to identify the word results corresponding to handwritten actions and transmit them back to the mobile device.

[0107] This application inputs the channel impulse response (CIR) feature into a deep neural network for handwriting recognition, including a deep feature extractor and a sequence information extractor.

[0108] like Figure 5 The diagram shows a schematic of a CNN network model. The main structure of the CNN network used in this application consists of convolutional layers, batch normalization layers, ReLU activation function layers, and pooling layers. Convolutional layers are responsible for extracting features, batch normalization layers adjust the extracted features to make their distribution more uniform, the rectified linear unit (ReLU) activation function filters the extracted features, retaining some effective features to accelerate network convergence and prevent some overfitting, and pooling layers compress the data. To reduce information loss during computation, the convolutional layers use 3x3 kernels with a stride of 1 between each kernel.

[0109] Since the amount of CIR data based on ultrasound is much larger than that of traditional passive sensing signal feature data, it requires deep neural networks to learn the features related to handwriting actions. Therefore, this application preferably uses a stack of six layers of the above structure to achieve a higher recognition accuracy on existing samples while extracting effective features, and at the same time retains as much information in the data as possible to improve the model's ability to recognize across environments and users.

[0110] like Figure 6 The diagram shows a schematic of a temporal modeler. This application uses a bidirectional LSTM combined with the CTC loss function to construct a temporal model of a handwritten word CIR input sequence. The use of LSTM here is closer to the field of speech recognition, treating words as a whole sequence and the letters that make up the words as elements of the sequence. The LSTM receives the high-level representation produced by the feature extractor, extracts temporally relevant information, reduces the amount of output data through a fully connected layer, and then obtains the probability of the letter corresponding to each unit through the SoftMax normalization function.

[0111] The CTC loss function addresses the situation where the output sequence length does not match the expected sequence length. In this application, the word length *l* is not uniform; however, the length of the LSTM output is taken as the maximum value MAX(l) of all lengths to accommodate all possible words. Therefore, multiple LSTM outputs may be equivalent to the same word. The CTC function summarizes the probabilities of these equivalent permutations, i.e., given a word *a*, all outputs π equivalent to *a*. The calculation process of the CTC loss function is as follows:

[0112]

[0113] Among them, B -1(a) is the inverse function. The positive function B(π) = a means that π can be equivalent to the word a, and the inverse function means all such combinations of π. Pr(a|x) means the probability that the result is equivalent to a given output x.

[0114] The resulting CTC loss is then fed back to all neural network layers, including CNN and LSTM, and the parameters of the network are optimized by the Adam optimizer to gradually learn to recognize the features of different handwritten letters.

[0115] In simple terms, the server-side is responsible for downsampling the ultrasonic audio files, calculating the CIR, and recognizing the results from the machine learning model. The feature extraction module before the neural network extracts the real and imaginary parts of the CIR and their corresponding differences as multi-dimensional features, which are then input into the neural network. Convolutional layers and activation functions are used for feature compression and high-dimensional feature extraction. The deep features of the handwritten action sequence extracted by the CNN are used to perform temporal modeling with LSTM (Long Short-Term Memory) to extract a series of probability prediction vectors in letter dimensions. During the training phase, the CTC loss function updates the model based on the prediction results, gradually improving the accuracy of handwriting recognition. Simultaneously, the pen-moving action is also input into the model as an action classification during this stage, enabling the model to segment long sequences.

[0116] Finally, the server completes the recognition of the handwriting action and sends the recognized words back to the mobile phone. The mobile phone's input method program receives the results via wireless network and inputs them into the corresponding input box on the mobile phone.

[0117] In some examples, the classifier has a certain probability of misidentification. According to our observation, a misidentified word is likely to be wrong in only one or two letter positions. Therefore, we built a comparison dictionary after the classifier, and then compared the word obtained by the probability vector with the dictionary. If the classification result is not in the dictionary, we use the edit distance algorithm to find the closest letter in the dictionary to complete the error correction. This can reduce the recognition bias of a few letters and improve the recognition accuracy. At the same time, it can automatically correct the user's typos.

[0118] In summary, this method improves the resolution and spatial perception capability of ultrasonic signals by extracting CIR features through demodulation and Pearson correlation coefficient algorithms. Simultaneously, for CIR (Channel Impulse Response) features, this method utilizes CNN (Convolutional Neural Network) to extract generalizable features in gesture recognition, and extracts the temporal relationships between handwritten letters through CTC (Connectivity-Temporal Classification) loss function and LSTM (Long Short-Term Memory) network. This allows for the association of different word samples through common letter members, improving the feature extraction efficiency of a small number of samples and enhancing the model's generalization ability. Furthermore, by treating the pen movement between different words as a separate recognition object, this application accurately segments continuous handwritten input without requiring segmentation algorithms.

[0119] This application establishes a system that uses the built-in microphone and speaker of a smartphone to recognize handwriting actions at a certain distance (within 30cm) based on ultrasound. The system targets all 26 letters of the English alphabet and English words. Through machine learning techniques combining CTC loss and LSTM, the system achieves real-time recognition of handwritten input, achieving a recognition accuracy of 94.16% for the 26 letters in a collected dataset of commonly used words.

[0120] The common word dataset includes 100 commonly used words, where the letter distribution probability is roughly the same as that in the dictionary set. Sample contributors include 9 volunteers, covering four different collection environments, totaling 10,200 word samples. Specifically, for each volunteer, the letter recognition accuracy and word recognition accuracy are both higher than 92.80% and 79.05%, respectively.

[0121] By employing ultrasound coupled with autocorrelation sequences, robustness to environmental changes is improved, achieving a letter recognition accuracy of 85.37% in cross-environment experiments, enabling training once and application in multiple environments. Furthermore, the machine learning model achieves a word spacing recognition efficiency of 97.02%, demonstrating excellent word segmentation capabilities and supporting continuous handwritten input recognition.

[0122] This application demonstrates real-time and accurate recognition of handwritten English words and letters in practical applications, achieving an overall accuracy rate of 94.16%. Recognition latency is divided into network transmission latency and action recognition latency. Under conditions of an NVIDIA GTX1080 GPU, an Intel Core i9-7900X CPU (3.30GHz), and 32MB of RAM, the recognition latency for the entire feature calculation and machine learning process is 30-50ms. Network latency is limited by public equipment, typically ranging from 50-60ms under the current common 100Mbps wireless network bandwidth. The overall recognition latency is 80ms-110ms, providing a good user experience in input method application scenarios.

[0123] In addition, this application proposes a new input method by combining ultrasound with handwriting recognition, which solves the problem of difficulty in segmenting long sequence inputs in practical applications and partially solves the cross-environment problem faced by passive sensing, demonstrating the great potential of ultrasonic passive recognition in the commercial field.

[0124] like Figure 7 The diagram shows a schematic of a server structure. As shown, the server 700 includes: a memory 710, a processor 720, and a communicator 730; the memory stores computer instructions; the processor executes the computer instructions to implement... Figure 3 The method described therein; the communicator is used to communicate with, for example... Figure 1 The mobile communication connection shown is used to receive ultrasonic audio files or send word results corresponding to the recognized handwriting actions.

[0125] In some embodiments, the number of memories 701 in the computer device 700 may be one or more, and the number of processors 702 may be one or more. Figure 7 Each example is taken as an instance.

[0126] In one embodiment of this application, the processor 702 in the computer device 700 will perform as follows: Figure 1 The steps described involve loading one or more instructions corresponding to the process of an application into memory 701, and then having processor 702 run the application stored in memory 701, thereby achieving the following: Figure 1 The functions of the system.

[0127] The memory 701 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. The memory 701 stores an operating system and operating instructions, executable modules or data structures, or subsets thereof, or extended sets thereof. The operating instructions may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic business functions and handling hardware-based tasks.

[0128] The processor 702 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0129] The communicator 703 is used to establish communication connections between the database access device and other devices (e.g., clients, read-write databases, and read-only databases). The communicator 703 may include one or more modules with different communication methods, such as a CAN communication module connected to a CAN bus. The communication connection can be one or more wired / wireless communication methods and combinations thereof. Communication methods include any one or more of the following: the Internet, CAN, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. For example, any one or more combinations of Wi-Fi, Bluetooth, NFC, GPRS, GSM, and Ethernet.

[0130] In some specific applications, the various components of the computer device 700 are coupled together through a bus system, which may include not only a data bus but also a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 All kinds of buses are referred to as bus systems.

[0131] This application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0132] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this invention should still be covered by the claims of this application.

Claims

1. A method of handwriting recognition based on ultrasound, characterized in that, Applied to a server, the method comprises: Obtaining an ultrasonic audio file uploaded by a mobile terminal and representing a handwriting action; the ultrasonic audio file is obtained by playing an ultrasonic audio signal modulated into an ultrasonic frequency band by a speaker of the mobile terminal, receiving the ultrasonic audio signal reflected by the handwriting action in the vicinity of the mobile terminal through a microphone, and packing the received ultrasonic audio signal; wherein the GTS original sequence is a GSM Training Sequence, i.e., a GSM training sequence; Obtaining a GTS handwriting sequence contained in the ultrasonic audio file, and calculating a channel impulse response feature representing the handwriting action based on the GTS handwriting sequence; the process comprises: downsampling the ultrasonic audio file and demodulating to obtain a continuous GTS handwriting sequence; and dividing the continuous GTS handwriting sequence into multiple GTS handwriting sequence segments. The continuous GTS handwriting sequence is divided into multiple GTS handwriting sequence segments, which comprises: Sliding a window on the continuous GTS handwriting sequence using the autocorrelation of the GTS sequence; calculating a Pearson correlation coefficient with the GTS original sequence respectively, and taking a position with a coefficient close to 1 or -1 as a cutting point to divide multiple GTS handwriting sequence segments for extracting channel impulse response features respectively; wherein the GTS original sequence is modulated into an ultrasonic frequency band by the mobile terminal in advance to generate an ultrasonic audio signal, and the ultrasonic audio signal is played to collect the GTS handwriting sequence representing the handwriting action reflected back; Extracting real and imaginary parts of the channel impulse response feature and difference data of the corresponding time dimension as multi-dimensional feature input into a neural network model to extract deep features associated with different words or letters in the handwriting action sequence; A time sequence model constructed based on a bidirectional LSTM and a CTC loss function is used to train a probability prediction vector corresponding to the word or letter dimension of the handwriting action sequence to identify the word result corresponding to the handwriting action and return to the mobile terminal; The time sequence model constructed based on the bidirectional LSTM and the CTC loss function is used to train the probability prediction vector corresponding to the word or letter dimension of the handwriting action sequence, which comprises: The LSTM reduces the number of output data through a fully connected layer, and obtains the probability of each unit corresponding to the letter through a SoftMax normalization function; Given a word a, all outputs π equivalent to a, and the calculation process of the CTC loss function is: where B -1 (a) is the inverse function, the positive function B(π) = a represents that π is equivalent to the word a, the inverse function represents all combinations of such π; Pr(a|x) represents the possibility of the result equivalent to a under the condition of the output result x.

2. The ultrasonic-based handwriting recognition method of claim 1, wherein, The calculation of the channel impulse response feature representing the handwriting action comprises: Based on the LTI system, the GTS handwriting sequence segment r(t) is converted by convolution of the GTS original sequence x(t) to obtain: r(t) = h(t) * x(t), wherein h(t) represents a channel impulse response signal; Assuming there are L propagation paths under multipath effect, the propagation delay of each path is τ i , and the signal strength parameter of each path is a i , then r(t) can be obtained as the superposition of L reflected signals: Since the signal sent by the speaker is discrete, h(t) is correspondingly represented in matrix form, i.e.: h(t) = [h0, h1,..., hN-1]T L ] T where h i is called the ith channel tap; Divide x(t) into P reference bits and L protection bits, and at the i-th moment, the GTS handwriting sequence segment is the result of the GTS original sequence i to L+i bits converted by multipath: r(i) = [x L+i ,x L+i-1 ,...,x i ]*h(t) + n; where n is a noise constant; After transformation, r(t) = Mx + N; M is a conversion matrix; At this point the h(t) computation is converted to an optimal estimation problem, finding the optimal estimate that minimizes the difference from r From the least squares estimation method we have:​ h(t) = (M H M) -1 M H r(t) ; where the superscript H denotes the conjugate operator.

3. A server, characterized by The server comprises a memory, a processor and a communicator; the memory is used to store computer instructions; the processor runs the computer instructions to realize the method in any one of claims 1 to 2; and the communicator is used to communicate with the mobile terminal to receive the ultrasonic audio file or send the word result corresponding to the recognized handwriting action.

4. An ultrasonic-based handwriting recognition system, characterized by, The system comprises: The mobile terminal is used to obtain the ultrasonic audio file corresponding to the reflected handwriting action; the ultrasonic audio file is obtained by playing the ultrasonic audio signal with the GTS original sequence modulated into the ultrasonic frequency band through the loudspeaker of the mobile terminal, receiving the ultrasonic audio signal reflected by the handwriting action near the mobile terminal through the microphone and then obtaining; wherein the GTS original sequence is GSM Training Sequence, i.e. GSM training sequence; The method for modulating the GTS original sequence into the ultrasonic frequency band comprises: determining the target bandwidth and target frequency band required for modulating the GTS original sequence; obtaining the frequency distribution of the GTS original sequence through Fourier transform according to the target bandwidth and the frequency of the audio file; obtaining the signal with the target bandwidth through inverse Fourier transform by extending the length of the GTS original sequence through the zero algorithm; up-sampling the signal into the target frequency band range through calculation with the sine wave with the specific frequency; wherein the specific frequency is the intermediate frequency of the target frequency band; wherein the determination method of the target frequency band is: obtaining the upper limit of the audio sampling rate of the mobile terminal; obtaining the upper limit of the signal frequency of the mobile terminal according to the Nyquist sampling theorem; determining the target frequency band according to the upper limit of the signal frequency and the lower limit frequency of the ultrasonic wave; and further used for uploading the ultrasonic audio file to the server through the communicator; The server is in communication connection with the mobile terminal, used to obtain the ultrasonic audio file uploaded by the mobile terminal and representing the handwriting action; obtaining the GTS handwriting sequence contained in the ultrasonic audio file according to the ultrasonic audio file, calculating the channel impulse response feature representing the handwriting action; extracting the real part, imaginary part and difference data of the corresponding time dimension of the channel impulse response feature as the multi-dimensional feature input into the neural network model to extract the deep features associated with different words or letters in the handwriting action sequence; constructing a time sequence model based on the bidirectional LSTM and CTC loss function to train the probability prediction vector of the word or letter dimension corresponding to the handwriting action sequence to identify the word result corresponding to the handwriting action, and returning the word result to the mobile terminal; The method for obtaining the GTS handwriting sequence contained in the ultrasonic audio file comprises: down-sampling the ultrasonic audio file and demodulating to obtain the continuous GTS handwriting sequence; and dividing the continuous GTS handwriting sequence into multiple GTS handwriting sequence segments; The method for dividing the continuous GTS handwriting sequence into multiple GTS handwriting sequence segments comprises: The GTS sequence is windowed by using the autocorrelation of the GTS sequence; the Pearson correlation coefficient is calculated with the original GTS sequence respectively, and the position with the coefficient close to 1 or -1 is taken as a cutting point to divide a plurality of GTS handwriting sequence segments for extracting channel impulse response features respectively; wherein the original GTS sequence is modulated to an ultrasonic frequency band by a mobile terminal in advance, and an ultrasonic audio signal is generated, and the GTS handwriting sequence containing the characteristic handwriting action is collected by playing and reflecting back; The time sequence model constructed based on the bidirectional LSTM and the CTC loss function is used to train a probability prediction vector of a word or letter dimension corresponding to a handwriting action sequence, and the time sequence model comprises: The LSTM reduces the number of output data through a full connection layer, and obtains the probability of each unit corresponding to a letter through a SoftMax normalization function; Given a word a, all outputs equivalent to a are pi, and the calculation process of the CTC loss function is as follows: where B -1 (a) is the inverse function, the positive function B(π) = a represents that π is equivalent to the word a, the inverse function represents all combinations of such π; Pr(a|x) represents the possibility of the result equivalent to a under the condition of the output result x. The mobile terminal is also used for receiving the word result returned by the server.

Images