Earphone inertial sensor-based real-time identification and reading method for lip language of Chinese aphonia people

Abstract

The invention discloses an earphone inertial sensor-based real-time lip language reading method for Chinese aphonia people, and belongs to the technical field of lip language translation. The method comprises the following steps: acquiring inertial data; segmenting syllable inertial data by using a lightweight method; fusing the accelerometer syllable inertial data and the gyroscope syllable inertial data by using a consistency method; recognizing syllable inertial data by using a dynamic time warping distance (DTW) method; and character selection is perfected and corrected by using context information of Chinese. According to the invention, inertial data expansion is realized through the converter, the data are processed, and noise data are eliminated; based on a normalized least mean square (NLMS) algorithm, the clean definition related signals are separated from the definition related signals interfered and influenced by the speaker, and the vibration influence generated by the earphone loudspeaker in the audio playing process is eliminated; the influence of diversity between the user and the equipment is inhibited, and the accuracy and accuracy of real-time lip language translation are ensured.

Description

Technical Field

[0001] The invention belongs to the technical field of lip reading translation, and in particular relates to a real-time lip reading method for Chinese voiceless people based on an earphone inertial sensor. Background Art

[0002] In the case of organic aphasia, dysphonia, post-laryngectomy and voice disorders, these users have complete pronunciation skills (such as lip and jaw movement), but they cannot enjoy the convenience of voice communication and voice interaction. Lip reading provides a friendly way to communicate without making sounds, leveraging the existing pronunciation process without additional learning costs. However, lip reading recognition is still very difficult, and solutions based on computer vision and acoustic sensing perform poorly in non-line-of-sight (NLOS) scenarios.

[0003] Ear-worn sensors offer a new solution for lip reading translation. Users naturally wear wireless headphones that are equipped with various sensors, including inertial sensors (accelerometers and gyroscopes) that can measure tiny vibrations. Therefore, the inertial sensors on the headphones can detect and identify the movements of the jaw and temporomandibular joint caused by lip reading.

[0004] Existing inertial sensing-based ear-worn devices are mainly developed on homemade prototypes rather than on commercial off-the-shelf (COTS) devices. Users need to pay extra for these dedicated peripherals or hardware modifications, which results in high costs and low customer willingness to purchase.

[0005] Therefore, how to use inertial sensors on COTS headphones to help Chinese-speaking people with voice loss achieve high-precision real-time lip translation is a technical problem that the present invention aims to solve. Summary of the invention

[0006] The purpose of the present invention is to provide a real-time lip reading method for Chinese voiceless people based on an earphone inertial sensor to solve the problems raised in the above background technology.

[0007] The object of the present invention is achieved by: a real-time lip reading method for Chinese voiceless people based on earphone inertial sensor, characterized in that the method comprises the following steps:

[0008] Step S1: Obtain the inertial data set in the registration phase and expand the inertial data set;

[0009] Step S2: using denoising technology to obtain error-free inertial data;

[0010] Step S3: segmenting syllable inertia data using a lightweight method;

[0011] The syllable inertia data includes accelerometer syllable inertia data and gyroscope syllable inertia data;

[0012] Step S4: fusing the accelerometer syllable inertial data and the gyroscope syllable inertial data using a consistency method;

[0013] Step S5: using the dynamic time warping distance DTW method to identify syllable inertial data according to the reference data pre-registered by the user;

[0014] Step S6: Using the context information of Chinese language to improve and correct the character selection.

[0015] Preferably, in step S1, the inertial data set in the registration phase is obtained and the inertial data set is expanded, and the specific operations are:

[0016] Step S1-1: using the inertial data collected by the headset inertial sensor;

[0017] Step S1-2: construct a converter model, and use the converter model to modify the inertial data of the source user and the silent pronunciation user to achieve the expansion of the inertial data;

[0018] The converter model includes an ID encoder for identifying individual features , content encoder for extracting content vectors and a decoder D for combining the identity and content of inertial data, ID encoder It includes two layers of LSTM with a feature dimension of 768 and a fully connected layer. The fully connected layer converts the output of the LSTM into the final prediction result; ID encoder The input data is variables collected from target speech-impaired users ,in, User IDs representing the lip reading user population, is the lip reading content vector collected from lip reading users, represents the inertial readings related to phonation collected from a population of lip readers;

[0019] The content encoder It includes three convolutional layers with a convolution kernel size of 5*1 and two layers of bidirectional long short-term memory network BLSTM with a feature dimension of 32, content encoder Input is the variables collected from the source user ,in, User IDs representing people with normal pronunciation function, is the speech content vector collected from normal people, represents the inertial readings related to articulation collected from a normal population;

[0020] The decoder D includes a concatenate layer, three first convolution layers with a convolution kernel size of 5*1, three layers of long short-term memory network units LSTM with a feature dimension size of 512, four second convolution layers with a convolution kernel size of 5*1, and a third convolution layer with a convolution kernel size of 5*1. The concatenate layer connects the outputs of the ID encoder and the content encoder together to merge different IMU data sources;

[0021] The first convolutional layer with three convolution kernels of size 5*1 and three layers of long short-term memory network units LSTM with feature dimension size of 512 are used to adjust the number of features and combine features;

[0022] The second convolution layer with four convolution kernels of size 5*1 is used to extract features, and the third convolution layer with a convolution kernel of size 5*1 is used for the final output adjustment.

[0023] Preferably, in step S1-2, the inertial data of the source user and the silent pronunciation are modified by using the converter model to achieve the expansion of the inertial data, and the specific operations are:

[0024] The inertial data with variable characteristics collected by the source users is represented as , the inertial data with variable characteristics collected by silent pronunciation users is expressed as ;

[0025] Source User and source user Speak the same thing , ID encoder Pre-training feature extraction , content encoder From the spectrogram and Features Constructing information bottlenecks in the concatenated features;

[0026] The decoder D follows the Auto VC principle under the loss function, that is:

[0027] min E CV ,D  L=& E [∥I MU ̂ S1→S2 -IM U S2 ∥ 2 2 ] &-λ E [|| E CV (I M ̂ U S1→S2 )- E CV (IM U S2 )| | 1 ] ;

[0028] in, To generate spectra; is the real spectrum; is the expected value, which represents the loss function is the average loss over all training samples; is the regularization parameter, which is used to balance the weights of the two parts in the loss function and controls the contribution of the regularization term to the total loss;

[0029] Real spectrum and generate spectra There is a high similarity between them, and the signal of inertial data is estimated from the spectrum using the Griffin-Lim algorithm.

[0030] Preferably, in step S2, a denoising technique is used to obtain error-free inertial data, specifically:

[0031] Step S2-1: using Wiener filtering to eliminate inherent noise; the inherent noise includes DC bias and additive white noise;

[0032] Step S2-2: using a deep regression model to remove inertial data interfered by sensors when the user moves; user movement includes movement of the user's body and head, mainly including walking, riding a bicycle, driving, and shaking the head;

[0033] Step S2-3: using a deep regression model to remove the influence of the user's lip movement on the inertial data; the collected inertial data corresponding to the user's lip movement includes a signal containing semantic information and a signal not containing semantic information, the signal containing semantic information is the required signal, and the signal not containing semantic information is a noise signal that needs to be removed, which is mainly divided into two categories: continuous movement and sudden movement, the representative action of continuous movement is chewing, and the representative action of sudden movement is yawning and coughing;

[0034] Step S2-4: using downsampling and an adaptive noise filter based on the normalized least mean square algorithm NLMS to remove the influence of the audio played by the earphone speaker on the inertial data;

[0035] According to the Shannon-Nyquist sampling theorem, the headphone audio with a sampling rate much higher than that of the inertial sensor is downsampled:

[0036] ;

[0037] in, Inertial sensor The downsampling noise of the axis, ; is the transfer function, , and Respectively represent the amplitude, frequency and phase of the audio;

[0038] An adaptive noise filter based on the normalized least mean square algorithm (NLMS) is used to separate the pronunciation-related inertial signals without interference from the pronunciation-related inertial signals affected by interference in real time, thereby eliminating the vibration effect generated by the headphone speaker during audio playback.

[0039] Preferably, in step S3, the syllable inertia data is segmented using a lightweight method, and the specific operation is:

[0040] Step S3-1: Use a double threshold scheme to remove irrelevant inertial data and retain the inertial data corresponding to the lip reading action; use the OTSU algorithm to calculate the threshold of the inertial data corresponding to the required lip reading action on each sensor respectively, and when the inertial data part on both sensors exceeds the threshold of each sensor, it is considered to be the inertial data related to the lip reading pronunciation;

[0041] Step S3-2: Extend the starting point and the end point of each recognition area forward and backward by 5 samples respectively to ensure that the pronunciation actions related to the entire syllable are covered;

[0042] Step S3-3: Segment the pronunciation action into syllables to obtain syllable inertia data.

[0043] Preferably, in step S4, the accelerometer syllable inertial data and the gyroscope syllable inertial data are fused using a consistency method, specifically:

[0044] Step S4-1: Reduce the dimension of the accelerometer syllable inertial data, the formula is as follows:

[0045] ;

[0046] in, Indicates the accelerometer The inertia reading of the axis, Indicates the accelerometer The inertia reading of the axis, Indicates the accelerometer Inertia readings of axes; is a symbol function, which means returning a symbol; represents the maximum value function;

[0047] To reduce the dimension of the gyroscope syllable inertial data, the formula is as follows:

[0048] ;

[0049] in, Indicates the gyroscope The inertia reading of the axis, Indicates the gyroscope The inertia reading of the axis, Indicates the gyroscope Inertia readings of axes;

[0050] Step S4-2: Use a point-by-point multiplier to fuse the two sources into a one-dimensional feature vector, the formula is as follows:

[0051] ;

[0052] in, is the one-dimensional accelerometer syllable inertial data; is the one-dimensional gyroscope syllable inertial data, is a point-by-point multiplier.

[0053] Preferably, in step S5, the syllable inertia data is recognized using the dynamic time warping distance Fast-DTW method according to the reference data pre-registered by the user, specifically:

[0054] The Fast-DTW method aligns the fused data in step S4 with the pre-registered reference database and ranks potential articulatory gestures according to the DTW distance, where a low distance indicates a high similarity between the reference and the gesture;

[0055] Step S5-1: Detect the pinyin of monosyllabic letters:

[0056] Single vowel , , as well as Selected; including the following:

[0057] Only appears in initials after;

[0058] Only appears in after;

[0059] Single vowel Only used alone;

[0060] Single vowel Only appears in middle;

[0061] Step S5-2: Identify the finals containing vowels;

[0062] Step S5-3: using the prior knowledge of the identified finals to identify the inertial data of the entire syllable, and realizing independent recognition of the initial consonant.

[0063] Preferably, the step S6 uses the context information of Chinese to improve and correct the selection of characters, specifically:

[0064] Step S6-1: Correcting the misrecognized syllables using context information;

[0065] The Chinese Pinyin initials are divided into seven categories. Denote; define transition probability ,in, For the initial consonant, is the final; the Bayesian equation is used to calculate the transmission probability:

[0066] ;

[0067] in, Indicated in Under the condition of finals Established The number of initial consonants;

[0068] Step S6-2: Consider the constraints of Chinese language and context analysis;

[0069] Step S6-3: Continuous translation based on minimum DTW distance and highest probability;

[0070] The lip sentences are translated continuously by calculating the minimum DTW distance and the highest transition probability; for each recognized syllable, the syllable with the minimum DTW distance and the highest transition probability is selected as the final result.

[0071] Preferably, the constraints of Chinese language and context analysis are considered in step S6-2, specifically:

[0072] The constraints include: feasibility constraints of syllable combinations, uneven frequency constraints of syllable usage, and frequent usage constraints of phrases;

[0073] Feasibility constraints of syllable combination: The specificity of the combination of initial syllable and final vowel in Chinese is reflected in:

[0074] Only appears in , , , and After waiting for a specific initial consonant, the conditional probability is generated ;

[0075] The uneven frequency of syllables: The frequency of syllables in Chinese is uneven, as shown in the following:

[0076] syllable Compare To be common, syllable and The four pronunciation gestures are similar but not used in Chinese;

[0077] Frequent usage constraint of phrases: Phrases are often used in Chinese instead of single characters. The context will affect the transition probability. It is necessary to consider the context information to adjust the transition probability. The Bayesian equation in step S6-1 is improved as follows:

[0078] ;

[0079] in, Represents a consonant The number of preceding or following letters, For the preceding letter, Followed by letters.

[0080] Compared with the prior art, the present invention has the following improvements and advantages:

[0081] 1. The inertial data is expanded through the converter, and the data is processed to eliminate noise data; the clean clarity-related signals and the clarity-related signals affected by the speaker's interference are separated in real time by the normalized least mean square algorithm NLMS, thereby eliminating the vibration effect of the headphone speaker during audio playback; at the same time, the influence of diversity between users and devices is suppressed, ensuring the correctness and accuracy of real-time lip reading translation.

[0082] 2. The method of the present invention provides real-time barrier-free communication and interactive services for users who cannot use voice, uses COTS headphones, does not require other peripherals, fully utilizes the limited capabilities of mobile devices, and realizes sensing based on inertial data. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 The figure is an overall flow chart of the method of the present invention.

[0084] Figure 2 This is the structure diagram of the converter model.

[0085] Figure 3 This is the energy distribution structure diagram of the inherent noise.

[0086] Figure 4 This is the change diagram of the accelerometer x-axis after removing the inherent noise.

[0087] Figure 5 This is the frequency response diagram of the experimental equipment.

[0088] Figure 6 This is a comparison chart of the accuracy of phrase recognition.

[0089] Figure 7 This is a comparison chart of syllable recognition accuracy. DETAILED DESCRIPTION

[0090] The present invention is further summarized below with reference to the accompanying drawings.

[0091] like Figure 1 As shown, a real-time lip reading method for Chinese voiceless people based on an earphone inertial sensor comprises the following steps:

[0092] Step S1: Obtain the inertial data set in the registration phase and expand the inertial data set;

[0093] Step S1-1: Use a COTS headset with an inertial sensor and a corresponding Apple phone as a data acquisition device;

[0094] Step S1-2: construct a converter model, and use the converter model to modify the inertial data of the source user and the silent pronunciation user to achieve the expansion of the inertial data;

[0095] The converter is expressed as follows:

[0096] ;

[0097] like Figure 2 As shown, the converter model includes an ID encoder for identifying individual features. , content encoder for extracting content vectors and a decoder D for combining the identity and content of inertial data, ID encoder It includes two layers of LSTM with a feature dimension of 768 and a fully connected layer. The fully connected layer converts the output of the LSTM into the final prediction result; ID encoder The input data is variables collected from target speech-impaired users ,in, User IDs representing the lip reading user population, is the lip reading content vector collected from lip reading users, represents the inertial readings related to phonation collected from a population of lip readers;

[0098] Content Encoder It includes three convolutional layers with a convolution kernel size of 5*1 and two layers of bidirectional long short-term memory network BLSTM with a feature dimension of 32, content encoder Input is the variables collected from the source user ,in, User IDs representing people with normal pronunciation function, is the speech content vector collected from normal people, represents the inertial readings related to articulation collected from a normal population;

[0099] The decoder D includes a concatenate layer, three first convolution layers with a convolution kernel size of 5*1, three layers of long short-term memory network units LSTM with a feature dimension size of 512, four second convolution layers with a convolution kernel size of 5*1, and a third convolution layer with a convolution kernel size of 5*1. The concatenate layer connects the outputs of the ID encoder and the content encoder together to merge different IMU data sources;

[0100] The first convolutional layer with three convolution kernels of size 5*1 and three layers of long short-term memory network units LSTM with feature dimension size of 512 are used to adjust the number of features and combine features;

[0101] The second convolution layer with four convolution kernels of size 5*1 is used to extract features, and the third convolution layer with a convolution kernel of size 5*1 is used for the final output adjustment.

[0102] The converter model is used to modify the inertial data of the source user and the silent utterance to achieve the expansion of the inertial data. The specific operations are as follows:

[0103] The inertial data with variable characteristics collected by the source users is represented as , the inertial data with variable characteristics collected by silent pronunciation users is expressed as ;

[0104] Source User and source user Speak the same thing , ID encoder Pre-training feature extraction , content encoder From the spectrogram and Features Constructing information bottlenecks in the concatenated features;

[0105] The decoder D follows the Auto VC principle under the loss function, that is:

[0106] min E CV ,D  L=& E [∥I MU ̂ S1→S2 -IM U S2 ∥ 2 2 ] &-λ E [|| E CV (I M ̂ U S1→S2 )- E CV (IM U S2 )| | 1 ] ;

[0107] in, To generate spectra; is the real spectrum; is the expected value, which represents the loss function is the average loss over all training samples; is the regularization parameter, which is used to balance the weights of the two parts in the loss function and controls the contribution of the regularization term to the total loss;

[0108] Real spectrum and generate spectra There is a high similarity between them, and the signal of inertial data is estimated from the spectrum using the Griffin-Lim algorithm;

[0109] The signal of inertial data is estimated from the spectrum using the Griffin-Lim algorithm, specifically:

[0110] Initialization: Read the amplitude spectrum calculated from the IMU data. Since the Griffin-Lim algorithm does not require the original phase information, a phase spectrum is randomly initialized;

[0111] Iterative process: Use the current phase spectrum and amplitude spectrum to reconstruct the audio signal through the inverse short-time Fourier transform (ISTFT);

[0112] Perform STFT again on the reconstructed audio signal to obtain new amplitude spectrum and phase spectrum;

[0113] Keep the original amplitude spectrum consistent with the new amplitude spectrum, and use the new phase spectrum to update the current phase spectrum;

[0114] Repeating iteration: Repeating the iterative process until a preset number of iterations is reached or the difference between the reconstructed audio signal spectrogram and the original spectrogram is small enough;

[0115] Termination condition: When the difference between the amplitude spectrum of the reconstructed signal and the target amplitude spectrum is less than a certain threshold, or when the maximum number of iterations is reached, the algorithm terminates;

[0116] Reconstructed signal: Finally, the final phase spectrum and target amplitude spectrum are used to perform an inverse short-time Fourier transform (ISTFT) to obtain the reconstructed inertial data signal.

[0117] Step S2: Use denoising technology to obtain error-free inertial data. The specific operations are as follows:

[0118] Step S2-1: Using Wiener filtering to eliminate inherent noise;

[0119] Intrinsic noise includes DC bias and additive white noise; the energy distribution of inherent noise is as follows: Figure 3 As shown, the horizontal axis in the figure represents the power spectrum density, and the unit is 10 -6 m 2 / s 4 Hz, the vertical axis represents the frequency, ranging from 0Hz to 12.5Hz;

[0120] The DC bias is expressed as:

[0121] DC bias ;

[0122] in, is the sampled value of the signal, is the total number of sampling points;

[0123] Subtract the calculated DC offset value from the original signal, expressed as:

[0124] ;

[0125] in, is the signal sampling value after debiasing;

[0126] In the frequency domain, the transfer function of the Wiener filter is multiplied by the Fourier transform of the debiased signal, and then the inverse Fourier transform is performed to obtain the filtered signal. The transfer function of the Wiener filter is It is expressed as:

[0127] ;

[0128] The filtered signal is expressed as:

[0129] ;

[0130] in, is the Fourier transform of the filtered signal, is the Fourier transform of the signal after debiasing. The change after removing noise is as follows Figure 4 shown.

[0131] Step S2-2: using a deep regression model to remove inertial data interfered by sensors when the user moves; user movement includes movement of the user's body and head, mainly including walking, riding a bicycle, driving, and shaking the head;

[0132] Step S2-3: using a deep regression model to remove the influence of the user's lip movement on the inertial data; the collected inertial data corresponding to the user's lip movement includes a signal containing semantic information and a signal not containing semantic information, the signal containing semantic information is the required signal, and the signal not containing semantic information is a noise signal that needs to be removed, which is mainly divided into two categories: continuous movement and sudden movement, the representative action of continuous movement is chewing, and the representative action of sudden movement is yawning and coughing;

[0133] The deep regression model includes a flattening layer to convert the raw input data into a one-dimensional array; two fully connected layers to learn the complex features of the input data; two random inactivation layers to randomly "inactivate" some neurons to reduce the risk of overfitting; a regression layer to output the results using a linear activation function;

[0134] Step S2-4: using downsampling and an adaptive noise filter based on a normalized least mean square (NLMS) algorithm to remove the influence of the audio played by the earphone speaker on the inertial data;

[0135] According to the Shannon-Nyquist sampling theorem, the headphone audio with a sampling rate much higher than that of the inertial sensor is downsampled:

[0136] ;

[0137] in, Inertial sensor The downsampling noise of the axis, ; is the transfer function, , and Represent the amplitude, frequency and phase of the audio respectively; for a given sensor, the transfer function is consistent, such as Figure 5 The figure below shows the frequency response of the experimental equipment. And use an adaptive noise filter. Based on the normalized least mean square algorithm NLMS, the adaptive noise filter can separate the pronunciation-related inertial signals without interference from the pronunciation-related inertial signals affected by interference in real time, so as to eliminate the vibration influence generated by the headphone speaker during audio playback.

[0138] Step S3: Segment the syllable inertia data using a lightweight method. The specific operations are as follows:

[0139] Step S3-1: Use a double threshold scheme to remove irrelevant inertial data and retain the inertial data corresponding to the lip reading action; use the OTSU algorithm to calculate the threshold of the inertial data corresponding to the required lip reading action on each sensor respectively, and when the inertial data part on both sensors exceeds the threshold of each sensor, it is considered to be the inertial data related to the lip reading pronunciation;

[0140] Calculate the histogram of each sensor signal strength and calculate the cumulative histogram of each signal strength value, expressed as ,in, is the signal strength level;

[0141] For each possible threshold , the signal strength is divided into two categories, including class 1 and class 2, class 1 is the signal strength is less than or equal to the threshold , the signal strength of class 2 is greater than the threshold , calculate the between-class variance :

[0142] σ between 2 (t)= P 0 (t)⋅ P 1 (t) P(t ) 2 ⋅[ μ 0 (t)- μ 1 (t) ] 2 ;

[0143] in: is the probability of class 1; is the probability of class 2; is the total probability, that is ; is the average signal strength of class 1; is the average signal strength of class 2.

[0144] Step S3-2: Extend the starting point and the end point of each recognition area forward and backward by 5 samples respectively to ensure that the pronunciation actions related to the entire syllable are covered;

[0145] Step S3-3: Segment the pronunciation action into syllables to obtain syllable inertia data.

[0146] According to the language structure of Chinese, the initial consonants are composed only of consonants, while the finals must begin with vowels. Vowels are generally the loudest sounds and are easiest to detect, so the first appearance of vowels is used as the entry point for syllable segmentation. At the same time, the pronunciation of vowels will cause significant changes in the inertial sensor readings. After finding the first peak that exceeds the threshold on the axis of the highest rotational change, the zero-crossing point before the peak is designated as the cutting point.

[0147] Step S4: using a consistency method to fuse the accelerometer syllable inertial data and the gyroscope syllable inertial data, specifically:

[0148] Step S4-1: Reduce the dimension of the accelerometer syllable inertial data, the formula is as follows:

[0149] ;

[0150] in, Indicates the accelerometer The inertia reading of the axis, Indicates the accelerometer The inertia reading of the axis, Indicates the accelerometer Inertia readings of axes; is a symbol function, which means returning a symbol; represents the maximum value function;

[0151] To reduce the dimension of the gyroscope syllable inertial data, the formula is as follows:

[0152] ;

[0153] in, Indicates the gyroscope The inertia reading of the axis, Indicates the gyroscope The inertia reading of the axis, Indicates the gyroscope Inertia readings of axes;

[0154] Step S4-2: Use a point-by-point multiplier to fuse the two sources into a one-dimensional feature vector, the formula is as follows:

[0155] ;

[0156] in, is the one-dimensional accelerometer syllable inertial data; is the one-dimensional gyroscope syllable inertial data, is a point-by-point multiplier.

[0157] Step S5: using the Fast-DTW method to identify syllable inertial data based on the reference data pre-registered by the user, specifically:

[0158] The Fast-DTW method aligns the fused data in step S4 with the pre-registered reference database and ranks potential articulatory gestures according to the DTW distance, where a low distance indicates a high similarity between the reference and the gesture;

[0159] Step S5-1: Detect the pinyin of monosyllabic letters:

[0160] Single vowel , , as well as Selected; including the following:

[0161] Only appears in initials after;

[0162] Only appears in after;

[0163] Single vowel Only used alone;

[0164] Single vowel Only appears in middle;

[0165] Step S5-2: Identify the finals containing vowels;

[0166] It is observed that each final has different clarity. Compared with the initial consonant, the final has the characteristics of long duration and high intensity. At the same time, the end of a pronunciation gesture may be affected by the subsequent syllables, resulting in distortion and overlap. The final segmentation result is matched with the final data of the reference, and the DTW distance between the pronunciation action and the reference signal is compared. The three finals with the shortest DTW distance are selected for the subsequent steps.

[0167] Step S5-3: using the prior knowledge of the identified finals to identify the inertial data of the entire syllable, and realizing independent recognition of the initial consonant.

[0168] The pronunciation of consonants is easily affected by the subsequent finals and cannot be distinguished based on inertial reading alone. The prior knowledge of the recognized finals is used to recognize the entire syllable, thereby achieving independent recognition of the initials; these indistinguishable initials are named visemes.

[0169] Step S6: Using the context information of Chinese to improve and correct the character selection, specifically:

[0170] Step S6-1: Correcting the misrecognized syllables using context information;

[0171] The Chinese Pinyin initials are divided into seven categories. Denote; define transition probability ,in, For the initial consonant, is the final; the Bayesian equation is used to calculate the transmission probability:

[0172] ;

[0173] in, Indicated in Under the condition of finals Established The number of initial consonants;

[0174] Step S6-2: Consider the constraints of Chinese language and context analysis, specifically:

[0175] The constraints include: feasibility constraints of syllable combinations, uneven frequency constraints of syllable usage, and frequent usage constraints of phrases;

[0176] Feasibility constraints of syllable combination: The specificity of the combination of initial syllable and final vowel in Chinese is reflected in:

[0177] Only appears in , , , and After waiting for a specific initial consonant, the conditional probability is generated ;

[0178] The uneven frequency of syllables: The frequency of syllables in Chinese is uneven, as shown in the following:

[0179] syllable Compare To be common, syllable and The four pronunciation gestures are similar but not used in Chinese;

[0180] Frequent usage constraint of phrases: Phrases are often used in Chinese instead of single characters. The context will affect the transition probability. It is necessary to consider the context information to adjust the transition probability. The Bayesian equation in step S6-1 is improved as follows:

[0181] ;

[0182] in, Represents a consonant The number of preceding or following letters, For the preceding letter, is followed by a letter; for example: Although ,but Followed by The probability will exceed .

[0183] Step S6-3: Continuous translation based on minimum DTW distance and highest probability;

[0184] The lip sentences are translated continuously by calculating the minimum DTW distance and the highest transition probability; for each recognized syllable, the syllable with the minimum DTW distance and the highest transition probability is selected as the final result.

[0185] In order to verify the feasibility and effect of the method of the present invention, the experimental results are presented through the evaluation of relevant performance indicators and the use effect; the experimental settings are as follows:

[0186] The experiment was conducted on two COTS headsets. By calling the application programming interface API, the headsets collected 25Hz inertial data and sent it to the paired mobile device in real time via Bluetooth and ran the written Mute-Aid application on it.

[0187] The converter model is trained on a server equipped with an Intel ( R ) Xeon ( R ) Silver 4210R CPU @ 2.40GHz and 2 Nvidia RTX 3090s; the lip readings are played in audio or converted to text via ASR;

[0188] The data collected in this experiment are mainly divided into two situations: the data collection of 7 voiceless subjects was carried out on Apple AirPods 3 and iPad 9 (iPadOS 17.5.1); the data collection of 26 voiced volunteers was carried out on Apple Wireless Headphones Pro 2 and iPhone 13 mini (iOS17.6.1), including inertial data of 33 participants in total, totaling more than 80,000 items;

[0189] like Figure 6As shown in the figure, since the daily usage frequency of phrases in Chinese is higher than that of single words, the experimental recognition task is set to 20 phrases, with an average recognition accuracy of 90.32%; the performance of the present invention on COTS earplugs is only 4.18% lower than the most advanced (SOTA) lip reading translator (94.5%), which relies on specialized sensors / equipment. In the present invention, the performance of each phrase is presented along the diagonal line, with 30% of the phrases having an accuracy rate of more than 95%, 45% of the phrases having an accuracy rate of 90%, 85% of the phrases having an accuracy rate ≥ 80%, and no phrase having an accuracy rate of less than 70%.

[0190] The ability of the method of the present invention to recognize syllables is evaluated. In the absence of contextual information, recognizing monosyllables is more challenging than recognizing disyllabic phrases; Figure 7 As shown in the figure, the method of the present invention achieved a top-1 accuracy of 41.28% and a top-3 accuracy of 92.67% in 2,500 characters containing 50 types of syllables. In addition, it maintained a top-1 accuracy of more than 70% and a top-3 accuracy of more than 95% in the recognition of finals and visemes, which shows that the method of the present invention can recognize both phrases and characters, and shows the potential of the present invention in continuous lip reading in practice.

[0191] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A real-time lip reading method for Chinese voiceless people based on earphone inertial sensor, characterized by: The method comprises the following steps: Step S1: Obtain the inertial data set in the registration phase and expand the inertial data set; Step S2: using denoising technology to obtain error-free inertial data; Step S3: segmenting syllable inertia data using a lightweight method; The syllable inertia data includes accelerometer syllable inertia data and gyroscope syllable inertia data; Step S4: fusing the accelerometer syllable inertial data and the gyroscope syllable inertial data using a consistency method; Step S5: using the dynamic time warping distance DTW method to identify syllable inertial data according to the reference data pre-registered by the user; Step S6: Using the context information of Chinese language to improve and correct the character selection.

2. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 1, characterized in that: In step S1, the inertial data set of the registration phase is obtained and the inertial data set is expanded. The specific operations are as follows: Step S1-1: using the inertial data collected by the headset inertial sensor; Step S1-2: construct a converter model, and use the converter model to modify the inertial data of the source user and the silent pronunciation user to achieve the expansion of the inertial data; The converter model includes an ID encoder for identifying individual features , content encoder for extracting content vectors and a decoder D for combining the identity and content of inertial data, ID encoder It includes two layers of LSTM with a feature dimension of 768 and a fully connected layer. The fully connected layer converts the output of the LSTM into the final prediction result; ID encoder The input data is variables collected from target speech-impaired users ,in, User IDs representing the lip reading user population, is the lip reading content vector collected from lip reading users, represents the inertial readings related to phonation collected from a population of lip readers; The content encoder It includes three convolutional layers with a convolution kernel size of 5*1 and two layers of bidirectional long short-term memory network BLSTM with a feature dimension of 32, content encoder Input is the variables collected from the source user ,in, User IDs representing people with normal pronunciation function, is the speech content vector collected from normal people, represents the inertial readings related to articulation collected from a normal population; The decoder D includes a concatenate layer, three first convolution layers with a convolution kernel size of 5*1, three layers of long short-term memory network units LSTM with a feature dimension size of 512, four second convolution layers with a convolution kernel size of 5*1, and a third convolution layer with a convolution kernel size of 5*1. The concatenate layer connects the outputs of the ID encoder and the content encoder together to merge different IMU data sources; The first convolutional layer with three convolution kernels of size 5*1 and three layers of long short-term memory network units LSTM with feature dimension size of 512 are used to adjust the number of features and combine features; The second convolution layer with four convolution kernels of size 5*1 is used to extract features, and the third convolution layer with a convolution kernel of size 5*1 is used for the final output adjustment.

3. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 2 is characterized by: In step S1-2, the inertial data of the source user and the silent pronunciation are modified by using the converter model to achieve the expansion of the inertial data. The specific operation is: The inertial data with variable characteristics collected by the source users is represented as , the inertial data with variable characteristics collected by silent pronunciation users is expressed as ; Source User and source user Speak the same thing , ID encoder Pre-training feature extraction , content encoder From the spectrogram and Features Constructing information bottlenecks in the concatenated features; The decoder D follows the Auto VC principle under the loss function, that is: ； in, To generate spectra; is the real spectrum; is the expected value, which represents the loss function is the average loss over all training samples; is the regularization parameter, which is used to balance the weights of the two parts in the loss function and controls the contribution of the regularization term to the total loss; Real spectrum and generate spectra There is a high similarity between them, and the signal of inertial data is estimated from the spectrum using the Griffin-Lim algorithm.

4. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 1, characterized in that: In step S2, a denoising technique is used to obtain error-free inertial data, specifically: Step S2-1: using Wiener filtering to eliminate inherent noise; the inherent noise includes DC bias and additive white noise; Step S2-2: using a deep regression model to remove inertial data interfered by sensors when the user moves; user movement includes movement of the user's body and head, mainly including walking, riding a bicycle, driving, and shaking the head; Step S2-3: using a deep regression model to remove the influence of the user's lip movement on the inertial data; the collected inertial data corresponding to the user's lip movement includes a signal containing semantic information and a signal not containing semantic information, the signal containing semantic information is the required signal, and the signal not containing semantic information is a noise signal that needs to be removed, which is mainly divided into two categories: continuous movement and sudden movement, the representative action of continuous movement is chewing, and the representative action of sudden movement is yawning and coughing; Step S2-4: using downsampling and an adaptive noise filter based on the normalized least mean square algorithm NLMS to remove the influence of the audio played by the earphone speaker on the inertial data; According to the Shannon-Nyquist sampling theorem, the headphone audio with a sampling rate much higher than that of the inertial sensor is downsampled: ； in, Inertial sensor The downsampling noise of the axis, ; is the transfer function, , and Respectively represent the amplitude, frequency and phase of the audio; An adaptive noise filter based on the normalized least mean square algorithm (NLMS) is used to separate the pronunciation-related inertial signals without interference from the pronunciation-related inertial signals affected by interference in real time, thereby eliminating the vibration effect generated by the headphone speaker during audio playback.

5. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 1, characterized in that: In step S3, the syllable inertia data is segmented using a lightweight method, and the specific operation is as follows: Step S3-1: Use a double threshold scheme to remove irrelevant inertial data and retain the inertial data corresponding to the lip reading action; use the OTSU algorithm to calculate the threshold of the inertial data corresponding to the required lip reading action on each sensor respectively, and when the inertial data part on both sensors exceeds the threshold of each sensor, it is considered to be the inertial data related to the lip reading pronunciation; Step S3-2: Extend the starting point and the end point of each recognition area forward and backward by 5 samples respectively to ensure that the pronunciation actions related to the entire syllable are covered; Step S3-3: Segment the pronunciation action into syllables to obtain syllable inertia data.

6. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 1, characterized in that: In step S4, the accelerometer syllable inertia data and the gyroscope syllable inertia data are fused using a consistency method, specifically: Step S4-1: Reduce the dimension of the accelerometer syllable inertial data, the formula is as follows: ； in, Indicates the accelerometer The inertia reading of the axis, Indicates the accelerometer The inertia reading of the axis, Indicates the accelerometer Inertia readings of axes; is a symbol function, which means returning a symbol; represents the maximum value function; To reduce the dimension of the gyroscope syllable inertial data, the formula is as follows: ； in, Indicates the gyroscope The inertia reading of the axis, Indicates the gyroscope The inertia reading of the axis, Indicates the gyroscope Inertia readings of axes; Step S4-2: Use a point-by-point multiplier to fuse the two sources into a one-dimensional feature vector, the formula is as follows: ； in, is the one-dimensional accelerometer syllable inertial data; is the one-dimensional gyroscope syllable inertial data, is a point-by-point multiplier.

7. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 1, characterized in that: In step S5, the syllable inertia data is recognized using the dynamic time warping distance Fast-DTW method according to the reference data pre-registered by the user, specifically: The Fast-DTW method aligns the fused data in step S4 with the pre-registered reference database and ranks potential articulatory gestures according to the DTW distance, where a low distance indicates a high similarity between the reference and the gesture; Step S5-1: Detect the pinyin of monosyllabic letters: Single vowel , , as well as Selected; including the following: Only appears in initials after; Only appears in after; Single vowel Only used alone; Single vowel Only appears in middle; Step S5-2: Identify the finals containing vowels; Step S5-3: using the prior knowledge of the identified finals to identify the inertial data of the entire syllable, and realizing independent recognition of the initial consonant.

8. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 1, characterized in that: The step S6 uses the context information of Chinese to improve and correct the selection of characters, specifically: Step S6-1: Correcting the misrecognized syllables using context information; The Chinese Pinyin initials are divided into seven categories. Denote; define transition probability ,in, For the initial consonant, is the final; the Bayesian equation is used to calculate the transmission probability: ； in, Indicated in Under the condition of finals Established The number of initial consonants; Step S6-2: Consider the constraints of Chinese language and context analysis; Step S6-3: Continuous translation based on minimum DTW distance and highest probability; The lip sentences are translated continuously by calculating the minimum DTW distance and the highest transition probability; for each recognized syllable, the syllable with the minimum DTW distance and the highest transition probability is selected as the final result.

9. The method for real-time lip reading of Chinese voiceless people based on earphone inertial sensor according to claim 8, characterized in that: The constraints of Chinese language and context analysis are considered in step S6-2, specifically: Constraints include: The feasibility constraints of syllable combinations, the uneven frequency constraints of syllable usage, and the frequent usage constraints of phrases; Feasibility constraints of syllable combination: The specificity of the combination of initial syllable and final vowel in Chinese is reflected in: Only appears in , , , and After waiting for a specific initial consonant, the conditional probability is generated ; The uneven frequency of syllables: The frequency of syllables in Chinese is uneven, as shown in the following: syllable Compare To be common, syllable and The four pronunciation gestures are similar but not used in Chinese; Frequent usage constraint of phrases: Phrases are often used in Chinese instead of single characters. The context will affect the transition probability. It is necessary to consider the context information to adjust the transition probability. The Bayesian equation in step S6-1 is improved as follows: ； in, Represents a consonant The number of preceding or following letters, For the preceding letter, Followed by letters.

Images