A method for recognizing speech content using an ear-worn motion sensor

Abstract

The application relates to a method for recognizing speech content by using an ear-mounted motion sensor and belongs to the technical field of mobile computing applications. Firstly, an ear-mounted motion sensor is used to collect the motion state signals of the ears of a wearer and cut speech segments. Three-axis ECDM signals are analyzed, feature signals related to speech content are extracted, and individual differences in the features are eliminated. Three-axis BCV signals are analyzed, feature signals related to speech content are extracted. Finally, the obtained ECDM feature signals and BCV feature signals are used to recognize speech content based on a convolutional neural network and a connectionist temporal classification. The application only relies on the motion sensor universally embedded on the ear-mounted device to recognize speech content, can reliably operate under complex environmental noise, and has the advantages of anti-interference, low cost and the like.

Description

A method for recognizing speech content using an ear-worn motion sensor Technical Field

[0001] This invention relates to a speech content recognition method, and more particularly to a method for recognizing speech content using an ear-worn motion sensor, for expanding human-computer interaction applications, and belongs to the field of mobile computing application technology. Background Technology

[0002] With the rapid development of IoT technology and the increasing importance of human-computer interaction, the interactive tasks faced by computers are becoming more diverse and complex, promoting the diversification of human-computer interaction technologies. Speech content recognition (SCR), due to its natural form and contactless operation, has become a crucial part of human-computer interaction technology. Traditional SCR technology records audio through a microphone, matches speech phonemes based on their time and frequency domain characteristics, and finally integrates them into speech content. However, microphone-recorded audio is sensitive to environmental noise, causing these methods to fail in noisy environments. Therefore, achieving reliable speech content recognition in complex noisy environments has become a research hotspot in related fields both domestically and internationally.

[0003] To address complex environmental noise, researchers have developed numerous noise filtering algorithms. However, these algorithms are highly specific and often only handle a single type of noise. Some methods utilize large-scale microphone arrays to record noisy audio and extract noise and speech signals based on blind source separation techniques. However, microphone arrays are costly, complex to deploy, and rely on prior knowledge of environmental noise, making them difficult to apply in everyday life.

[0004] In addition, some methods use cameras to capture lip movements and then identify speech content. However, these camera-dependent methods pose a risk of privacy breaches and have stringent testing requirements, such as unobstructed faces and sufficient lighting. Other methods utilize millimeter-wave signals, Wi-Fi signals, or other radio frequency signals to capture lip and throat movements during speech to identify speech content. However, these methods are susceptible to multipath effects caused by body movement and generally require users to remain still during measurement, severely limiting user behavior. Finally, some methods use dedicated laryngeal bone conduction microphones to record bone conduction audio during speech to identify speech content. However, wearing a microphone in the throat presents numerous inconveniences in daily life, making large-scale adoption difficult.

[0005] In summary, existing methods have various shortcomings, and new methods are urgently needed to overcome their limitations. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems and defects of existing speech content recognition methods in complex environmental noise, and to creatively propose a method for recognizing speech content using an ear-worn motion sensor, which can accurately recognize the wearer's speech content in complex environmental noise.

[0007] With the continuous development of embedded and sensor technologies, wearable devices (such as smart headphones and smart hearing aids) have widely deployed motion sensors such as accelerometers and gyroscopes to realize functions such as wear monitoring and touch interaction, which significantly improves the sensing capabilities of wearable devices and provides strong support for sensing the wearer's voice content.

[0008] The method for recognizing speech content using an ear-worn motion sensor proposed in this invention is based on the following principle: During speech production, on the one hand, the vibration of the vocal cords is transmitted to the ear via the mandible and muscle tissue, which can be captured by the motion sensor of the ear-worn device and is called bone conduction vibration (BCV). On the other hand, the movement of organs such as the tongue and lips during speech causes deformation of the ear canal contour, which can also be captured by the motion sensor of the ear-worn device and is called ear canal dynamic motion (ECDM). The characteristics of BCV and ECDM depend on the speech content; therefore, analyzing BCV and ECDM can extract a large amount of information related to the speech content, ultimately recognizing the wearer's speech content.

[0009] The objective of this invention is achieved through the following technical solutions.

[0010] A method for recognizing speech content using an ear-worn motion sensor includes the following steps:

[0011] Step 1: Use an ear-worn motion sensor to collect motion signals (acceleration signals) from the wearer's ear and segment the speech.

[0012] The purpose of this step is to save computing resources by monitoring and segmenting speech-containing segments based on the characteristic of drastic changes in acceleration signals during speech.

[0013] Specifically, it includes the following steps:

[0014] Step 1.1: Use a filter to process the acquired triaxial acceleration signal and extract the triaxial bone conduction vibration (BCV) signal and the triaxial ear canal dynamic motion (ECDM) signal respectively.

[0015] Step 1.2: Analyze the triaxial BCV signal and triaxial ECDM signal obtained in Step 1.1, and use a threshold-based method to detect the start and end times of the speech segments and segment the speech segments.

[0016] Step 2: Analyze the triaxial ECDM signal, extract feature signals related to speech content, and eliminate individual differences in the features.

[0017] The purpose of this step is to remove noise from the motion state signal, extract feature signals related to the speech content, and reconstruct the feature signals to eliminate the influence of individual differences, thus preparing for the subsequent realization of user-independent speech content recognition.

[0018] Specifically, it includes the following steps:

[0019] Step 2.1: Use a deep regression network to process the triaxial ECDM signal in the speech segment obtained in Step 1.2 to remove noise caused by body movement.

[0020] Step 2.2: Analyze the clean three-axis ECDM signal obtained in Step 2.1 and extract the ECDM signal related to the speech content.

[0021] Step 2.3: Reconstruct the waveform of the ECDM signal extracted in Step 2.2 to eliminate the influence of individual differences.

[0022] Step 3: Analyze the triaxial BCV signal and extract feature signals related to the speech content.

[0023] The purpose of this step is to remove noise from the BCV and extract feature signals related to the speech content, in order to prepare for subsequent implementation of user-independent speech content recognition by combining data-driven methods.

[0024] Step 3.1: Process the triaxial BCV signal in the speech segment obtained in step 1.2 to eliminate noise caused by the vibration of the ear-worn device.

[0025] Step 3.2: Analyze the clean BCV signal obtained in Step 3.1 and extract the BCV signal related to the speech content.

[0026] Step 4: Using the ECDM and BCV signals obtained in steps 2.3 and 3.2, identify speech content based on Convolutional Neural Network (CNN) and Connectionist Temporal Classification (CTC).

[0027] Thus, from step 1 to step 4, a speech content recognition method using an ear-worn motion sensor has been implemented.

[0028] Beneficial effects

[0029] The method of the present invention has the following advantages compared with existing authentication technologies:

[0030] 1. This invention relies solely on motion sensors commonly embedded in ear-worn devices to recognize speech content, enabling reliable operation even in complex and noisy environments, and offering advantages such as anti-interference and low cost.

[0031] 2. This invention designs a noise cancellation method using pitch tracking, which can accurately separate noise caused by body movement from BCV signals.

[0032] 3. This invention designs a signal reconstruction method based on Procrustes transform, which can effectively reduce the instability of ECDM feature signals caused by individual differences, and is one of the foundations for realizing user-independent speech content recognition.

[0033] 4. This invention designs a feature signal generation algorithm that uses normal speech signals to generate BCV feature signals containing individual difference information, which is one of the foundations for realizing user-independent speech content recognition.

[0034] 5. This invention designs a deep learning model based on CNN and CTC, which can achieve accurate word-level speech content recognition independently of the user. Attached Figure Description

[0035] Figure 1 is a schematic diagram of the method of the present invention;

[0036] Figure 2 is an example diagram of the speech segment recognition method of the present invention;

[0037] Figure 3 is a comparison diagram of the ECDM feature signal Y-axis data before and after conversion according to the present invention;

[0038] Figure 4 is a flowchart of the time-pitch tracing process for extracting BCV signals according to the present invention;

[0039] Figure 5 is a diagram of the deep neural network structure built by the present invention based on CNN and CTC;

[0040] Figure 6 shows the system performance obtained from tests conducted by 14 volunteers in this invention.

[0041] Figure 7 shows the performance of the present invention in speech content recognition at different sampling rates;

[0042] Figure 8 shows the system performance of the present invention under different loudness levels;

[0043] Figure 9 shows the system performance of the present invention when using different ear-worn device prototypes. Detailed Implementation

[0044] The method of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] As shown in Figure 1, a method for recognizing speech content using an ear-worn motion sensor includes the following steps:

[0046] Step 1: Use an ear-worn motion sensor to collect motion signals (acceleration signals) from the user's ear and segment the speech.

[0047] The motion state signal includes the changes in acceleration of the accelerometer along the X, Y, and Z axes, which are called triaxial acceleration signals. These changes are then converted into electrical signal data.

[0048] Step 1.1: Use a filter to process the acquired triaxial acceleration signal and extract the triaxial BCV signal and triaxial ECDM signal respectively.

[0049] Due to the filtering effect of sound propagating within the human body, the BCV signal frequency range acquired by the ear-worn device is mainly distributed above 50Hz. However, the slower movement of organs such as the tongue and lips results in the EDCM signal frequency range acquired by the ear-worn device being mainly distributed below 50Hz. Therefore, the acquired triaxial acceleration signals can be processed using a high-pass filter and a low-pass filter with a cutoff frequency of 50Hz, respectively. The components with frequencies above 50Hz are the triaxial BCV signal, and the components below 50Hz are the triaxial ECDM signal.

[0050] Because the acceleration signals acquired by the ear-worn device are generally affected by body motion interference (the frequency range is mainly below 50Hz) and vibration interference caused by the music played by the ear-worn device itself (the frequency range is mainly below 50Hz), the triaxial BCV signal obtained in this step still contains ear-worn device vibration noise, and the triaxial ECDM signal obtained still contains body motion noise, which urgently needs to be removed.

[0051] Step 1.2: Analyze the triaxial BCV signal and triaxial ECDM signal obtained in Step 1.1, use a threshold-based method to monitor the start and end times of speech segments, and segment the speech segments.

[0052] Because the amplitudes of the three-axis BCV and three-axis ECDM signals are higher when the wearer speaks and lower when the wearer is not speaking, speech signal segments and non-speech signal segments are distinguished by detecting the amplitude values ​​of these signals. Since the coordinate system of the accelerometer in the ear-worn device rotates under different wearing conditions, the beginning and end of each speech signal segment may correspond to peaks or troughs in the BCV and ECDM signals on each axis. To solve this problem, this invention introduces a linear BCV signal, which is the half-power of the sum of the squares of the X, Y, and Z axes of the three-axis BCV signal. Similarly, a linear ECDM signal is introduced, which is the half-power of the sum of the squares of the X, Y, and Z axes of the three-axis ECDM signal.

[0053] To identify the start and end of a speech segment, a threshold T0 is set and compared with the linear BCV signal and the linear ECDM signal, as shown in Figure 2. When both the linear BCV signal and the linear ECDM signal exceed the threshold simultaneously, it is considered the start of a speech segment; when both are below the threshold simultaneously, it is considered the end of a speech segment. Based on the start and end times of the speech segment, the triaxial BCV signal and the triaxial ECDM signal are segmented.

[0054] Since body motion noise and ear-worn device vibration noise cannot cause the linear BCV signal and linear ECDM signal to exhibit high amplitude simultaneously, the above method can effectively identify the beginning and end of speech segments without being affected by body motion noise and ear-worn device vibration noise.

[0055] Preferably, the threshold T0 is set to 0.2 times the maximum value of the linear BCV signal and the linear ECDM signal. However, other settings within the range of [0.1, 0.5] times are also within the scope of protection of this invention.

[0056] Step 2: Analyze the triaxial ECDM signal, extract feature signals related to speech content, and eliminate individual differences in the features.

[0057] Step 2.1: Use a deep regression network to process the triaxial ECDM signal in the speech segment obtained in Step 1.2 to remove noise caused by body movement.

[0058] Users wearing ear-worn devices inevitably move their bodies while speaking, resulting in motion noise still being superimposed on the triaxial ECDM signal segment obtained in step 1.2. To address this, this invention constructs a deep regression network consisting of an input layer, a fully connected layer, and an output layer. The input layer processes the noise-affected triaxial ECDM signal, calculating the spectral density of its X, Y, and Z data using short-time Fourier transform and inputting this data to the fully connected layer. The fully connected layer is used to eliminate motion noise. The output layer uses inverse short-time Fourier transform to restore the noise-free triaxial spectral density to a clean triaxial ECDM signal.

[0059] During the deep regression network training phase, acceleration signals during speech were pre-collected using an ear-worn device in a noise-free environment, and acceleration signals during daily activities such as walking and running were collected in a non-speaking environment. A noise-free, clean three-axis ECDM signal and a three-axis body motion noise signal were obtained using the method in step 1. A three-axis ECDM signal interfered with by body motion noise was artificially synthesized by linearly superimposing body motion noise of [0.1, 1] times onto the clean three-axis ECDM signal. The artificially synthesized noisy three-axis ECDM signal was used as the input signal to the network input layer, and the network parameters were optimized by minimizing the mean square error between the output signal of the output layer and the corresponding clean three-axis ECDM signal.

[0060] During the usage phase, the noisy triaxial ECDM signal obtained in step 1.2 is input into the trained deep regression neural network, and the output is the clean triaxial ECDM signal.

[0061] Step 2.2: Analyze the clean three-axis ECDM signal obtained in Step 2.1 and extract ECDM feature signals related to the speech content.

[0062] To extract the ear canal contour deformation caused by the movement of organs such as the tongue and lips during speech, the upper envelope E of the three-axis ECDM signal X, Y, and Z axis data was extracted respectively. x E y E z As a characteristic signal of EDCM, E = {E} x E y E z}

[0063] Step 2.3: Process the ECDM feature signals extracted in Step 2.2 and reconstruct their waveforms to eliminate the influence of individual differences.

[0064] Individual differences in ear canal contours and tongue and lip movements during speech lead to variations in the ECDM feature signal when individuals speak the same content, affecting speech content recognition accuracy. Analysis shows that individual differences primarily cause variations in the amplitude of the ECDM feature signal. Therefore, this invention proposes a method based on Procrustes transform to modify the amplitude of the ECDM feature signal and eliminate the influence of individual differences. The specific steps are as follows:

[0065] First, an alignment method based on dynamic time warping is used to stretch the X, Y, and Z axis data of ECDM feature signals from multiple users to the same data length, and then the average signal is calculated. Where n is the number of users. This represents the average signal along the X-axis of the ECDM characteristic signals after stretching by n users. These represent the average signals of the ECDM characteristic signals after stretching by n users, on the Y and Z axes, respectively.

[0066] Then, the Procrustes transform is performed on the current ECDM feature signal E to be converted. This process involves rotation, scaling, and translation. The result of the transform is... for:

[0067]

[0068] Where H is the rotation matrix, α is the scaling factor, β is the translation factor, and α and β are both constants.

[0069] right Perform singular value decomposition: U and V are orthogonal, Γ is a diagonal matrix, and T denotes the matrix transpose. The rotation matrix H passes through H = UV. T Calculations are performed. Furthermore, the least squares method is used to search for reasonable values ​​of α and β, ensuring that all transformed ECDM feature signals have the minimum mean square error.

[0070] Figures 3(a) and (b) compare the trajectories of the ECDM feature signals along the Y-axis when four volunteers spoke the same content. It can be observed that the converted ECDM feature signals have similar trajectories for the same speech content, and the influence of individual differences is effectively eliminated.

[0071] Step 3: Analyze the triaxial BCV signal and extract feature signals related to the speech content.

[0072] Step 3.1: Process the triaxial BCV signal in the speech segment obtained in step 1.2 to eliminate noise caused by the vibration of the ear-worn device.

[0073] The ear-worn device is primarily used for audio playback, causing structural vibrations that interfere with the acceleration signal. This results in noise from the ear-worn device vibration still being superimposed in the triaxial BCV signal segment obtained in step 1.2. Since each user's voice has a relatively constant fundamental tone over a short period, a pure BCV signal is extracted by eliminating noise caused by ear-worn device vibration through tone tracking.

[0074] As shown in Figure 4, pitch tracing includes signal decomposition, channel aggregation, pitch tracing, and signal synthesis.

[0075] First, signal decomposition is performed. Specifically, a 64-channel Gammatone filter can be used to process the triaxial BCV signal segment obtained in step 1.2. The X, Y, and Z axis data are processed in the same way. The frequency bands of each filter channel overlap and the center frequency is uniformly distributed in the range of 50-400Hz. The bandwidth of each channel is 24.7 (4.37×f+1), where f is the center frequency. A sliding window is applied to analyze the filtered signal. Specifically, the window size is 20ms and the sliding step size is 10ms. The signal of the c-th filter channel in the w-th window is represented as e. c,w (t).

[0076] Then channel aggregation is performed. Specifically, if the signals e of adjacent sub-channels... w,c (t) and e w,c+1 (t) If they contain the same type of signal source (such as the wearer's voice or the vibration noise of the ear-worn device), then the two will have a high correlation. Therefore, the correlation coefficient S between adjacent channels is defined. c,w The following is how e is quantified. w,c(t) and e w,c+1 Correlation of (t):

[0077]

[0078] in, A c,w (τ) represents the signal e w,c The autocorrelation coefficient of (t) when the time lag is τ. For the expectation operator, For signal e w,c The mean of (t), Let A be the autocorrelation coefficient. c,w The mean of (τ). Compare with S c,w A threshold T1 is used; if the threshold is exceeded, two channels are aggregated into one channel. Preferably, T1 is set to 0.985, but other settings within the range of [0.5-0.99] are also within the scope of this invention. After iteratively processing all channel signals within the window, several channels are obtained.

[0079] Then, taking advantage of the characteristic that each user's voice has a constant pitch over a short period of time, the signals within a sliding window with the same pitch are tracked. Specifically, the c-th channel signal e within the sliding window w is calculated. w,c The fundamental period of (t), that is, let Time lag value for obtaining the maximum value In addition, the global pitch period of the original BCV signal within the sliding window w is calculated, i.e., let Time lag value for obtaining the maximum value Compare the fundamental period of each channel signal within the sliding window. With global pitch period If the difference is less than the threshold T2, the channel signal originates from the wearer's voice signal and is retained for synthesizing a clean BCV signal. Preferably, T2 is set to 0.985, but other settings within [0.6, 1) are also within the scope of this invention. To avoid erroneous tracking, segments shorter than 50ms can be discarded.

[0080] Finally, a pure BCV signal is synthesized. Specifically, the signal within each sliding window is processed continuously, the voice signal originating from the wearer is separated based on the global pitch period, the separated signal is processed using an inverse Gammatone filter bank, and then all channels are summed to obtain a pure BCV signal.

[0081] Step 3.2: Analyze the clean BCV signal obtained in Step 3.1 and extract feature signals related to the speech content.

[0082] Specifically, a short-time Fourier transform is applied to the X, Y, and Z axis data of the clean BCV signal obtained in step 3.1 to extract the spectral density as the BCV feature signal. This BCV feature signal can well represent the speech content information and is robust to individual differences, eliminating the need for further elimination of individual differences.

[0083] Step 4: Using the ECDM and BCV feature signals obtained in steps 2.3 and 3.2, recognize speech content based on CNN and CTC.

[0084] A deep neural network is constructed based on CNN and CTC, including a feature signal processor, a feature fusion unit, and a speech content recognizer. The network structure is shown in Figure 5.

[0085] Since BCV and ECDM feature signals have different physical meanings and convey different information, two sub-networks are constructed in the feature signal processor to process them separately. Each sub-network consists of multiple convolutional layers, max pooling layers, batch normalization (BN) layers, dropout layers, and flattening layers. The convolutional layers extract implicit speech content information, the max pooling layers perform subsampling and nonlinear compression, the BN layers further improve model performance and stability, the dropout layers prevent overfitting, and finally, the flattening layers combine the obtained information to generate a one-dimensional latent feature. The BCV sub-network takes the BCV feature signal as input. The temporal BCV feature signal is first processed using a 2-second sliding window with a 1-second stride. The convolutional layers in the BCV sub-model use 32, 64, 128, and 128 convolutional kernels, respectively, with each kernel having a size of 3×3. After processing by the BCV sub-model, the output is a one-dimensional feature vector Ψ containing speech content information. BCV The ECDM subnetwork takes the ECDM feature signal as input. The temporal ECDM feature signal is first processed using a 2-second sliding window with a 1-second stride. The ECDM subnetwork uses convolutional kernels of sizes 1×9 and 1×3, respectively, and outputs a one-dimensional feature vector Ψ containing speech content information. ECDM .

[0086] The feature fusion unit combines the feature vectors Ψ output by the ECDM subnetwork and the BCV subnetwork in the feature signal processor. ECDM and Ψ BCV By using bilinear pooling to fuse features into a single one-dimensional feature, the feature dimensions of BCV and ECDM are effectively reduced, providing a joint representation space for these two different feature types. The bilinear pooling process involves computing the outer product of the latent representations of the ECDM feature vector and the BCV feature vector. in For the Kronecker integrator, This is a matrix vectorization operator that transforms a matrix into a column vector using a linear transformation. Then, the L2 norm is applied to Ψ to obtain the final fused features.

[0087] Finally, the speech content recognizer classifies the fused features, using a softmax layer to obtain the probability that the current input is a certain word, and the word p with the highest probability is selected. t This is the recognition result. Because a sliding window is used to analyze the data during the recognition process, the recognition results often contain repeated words, making it impossible to correspond to the actual speech content. Therefore, the CTC layer is used to process the sequence y = y1, ..., y1, which contains repeated words accumulated by the softmax layer over a period of time. t ,…,y T To obtain the final identified word sequence y * Define CTC loss as:

[0088] Loss CTC =-ln(P(y) * |y)) (3)

[0089] Where P(*|*) represents the conditional probability operator.

[0090] During the training phase of a deep neural network, y is known. * The parameters of the deep learning model are updated iteratively using steps 1 to 4 to obtain y, and the loss is minimized. CTC It can train the constructed deep neural network.

[0091] Example verification

[0092] To verify the performance of this method, three pairs of ear-worn devices with different structures, each embedded with a motion sensor, were used. A total of 14 volunteers (7 men and 7 women) were recruited to participate in the experiment. During the experiment, each volunteer wore the prototype ear-worn device according to their natural habits and read sentences from the LibriTTS corpus. Simultaneously, a microphone 30cm away from the volunteers was used to record the speech audio to obtain the actual speech content and analyze the loudness. A total of 2080 sentences were collected for evaluation.

[0093] Precision, recall, and detection rate are used to evaluate system performance. Precision is defined as the ratio of correctly predicted words to all words correctly predicted as label A; recall is defined as the ratio of correctly predicted words to all words belonging to label A; and detection rate is defined as the ratio of the number of correctly identified words to the total number of predicted words in the sequence.

[0094] First, the overall performance of the proposed method was tested. A user-independent deep neural network was trained using data from 13 volunteers, and tested using data from one remaining volunteer. Experiments validated all 14 training and test set combinations, achieving an average precision of 86.18%, an average recall of 85.15%, and an average detection rate of 95.38%. Figure 6 illustrates the precision, recall, and detection rate for each training and test set combination. It can be observed that 9 out of the 14 volunteers achieved precision, recall, and detection rates exceeding 85%, while the remaining volunteers achieved these metrics above 70%. These results demonstrate that the proposed method can be used for user-independent speech content recognition, and users do not need to provide training data before using the method.

[0095] Next, the performance of this invention was tested at different sampling rates. All volunteers wore the prototype ear-worn device, and ear acceleration signals were collected at 400Hz, 600Hz, 800Hz, and 1000Hz during speech. Figure 7 shows the performance of speech content recognition at different sampling rates. As the sampling rate increased from 400Hz to 800Hz, accuracy, recall, and detection rate all improved significantly. When the sampling rate increased to 1000Hz, no significant change was observed in accuracy, recall, and detection rate. The results show that an 800Hz sampling rate is sufficient to capture most of the speech content information and achieve accurate speech content recognition. With the rapid development of embedded and sensor technologies, some commercial ear-worn devices already have accelerometer sensors that support an 800Hz sampling rate, providing a favorable technical platform for this invention.

[0096] Furthermore, the performance of this invention was tested under varying loudness levels of the wearer's speech, demonstrating its good performance across a range of loudnesses. Since the average loudness of human speech ranges from 55dB to 65dB, all volunteers spoke at loudness levels ranging from 40dB to 70dB (calculated based on audio signals recorded by a microphone 30cm away from the volunteer) while wearing the earpiece to record acceleration signals. Figure 8 shows the accuracy, recall, and detection rate of this invention at different speech loudness levels. The system performance continuously improves with increasing speech loudness. At a speech loudness of 55dB-65dB, the average accuracy of this invention was 94.70%, the recall was 93.07%, and the detection rate was 92.54%. This result demonstrates that this invention is capable of handling speech of varying loudness.

[0097] Finally, the performance of this invention was tested using different ear-worn devices. Data collected from two ear-worn devices was used to train the deep neural network, and data from the remaining ear-worn device was used for testing. All three training and test set combinations are denoted as Case 1, Case 2, and Case 3, respectively. Alternatively, a dataset was constructed using data collected from all three ear-worn devices, with 70% of the data used to train the deep neural network and the remaining 30% used to test the model; this is denoted as Case 4. Figure 9 shows the accuracy, recall, and detection rate in the four cases. The accuracy, recall, and detection rate in Cases 1 to 3 are slightly lower, indicating that the deep neural network trained on data from one ear-worn device prototype cannot be generalized to the other two ear-worn devices. Case 4 achieves better accuracy, recall, and detection rate than Cases 1 to 3. This result demonstrates that by continuously updating the dataset with data from different ear-worn devices, this method can overcome the differences between ear-worn devices and achieve accurate speech content recognition.

[0098] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is used to explain the present invention. It is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for speech content recognition using an ear-worn motion sensor, characterized in that, Includes the following steps: Step 1: Use an ear-worn motion sensor to acquire motion signals from the wearer's ear and segment speech segments, including: Step 1.1: Process the acquired triaxial acceleration signals and extract triaxial BCV and triaxial ECDM signals respectively; Step 1.2: Analyze the triaxial BCV and triaxial ECDM signals obtained in Step 1.1, and use a threshold-based method to detect the start and end times of speech segments and segment them; wherein, a threshold T0 is set and compared with the linear BCV and linear ECDM signals. When both the linear BCV and linear ECDM signals exceed the threshold, it is considered the start of a speech segment; when both are below the threshold, it is considered the end of a speech segment; the triaxial BCV and triaxial ECDM signals are segmented according to the start and end times of the speech segment; Step 2: Analyze the triaxial ECDM signal, extract feature signals related to the speech content, and eliminate individual differences in the features, including: Step 2.1: Utilize... Step 1.2: Process the ECDM signal in the speech segment obtained in step 1.2 using a deep regression network to remove noise caused by body movement; Step 2.2: Analyze the clean three-axis ECDM signal obtained in step 2.1 to extract ECDM feature signals related to the speech content; Step 2.3: Process the ECDM feature signals extracted in step 2.2 and reconstruct their waveforms to eliminate the influence of individual differences; Step 3: Analyze the three-axis BCV signal to extract feature signals related to the speech content, including: Step 3.1: Process the three-axis BCV signal in the speech segment obtained in step 1.2 to eliminate noise caused by the vibration of the ear-worn device; Step 3.2: Analyze the clean BCV signal obtained in step 3.1 and extract BCV feature signals related to the speech content; Step 4: Use the ECDM feature signals and BCV feature signals obtained in steps 2.3 and 3.2 to identify the speech content based on convolutional neural networks and connectionist spatiotemporal classification.

2. The speech content recognition method using an ear-worn motion sensor as described in claim 1, characterized in that, In step 1.1, the acquired triaxial acceleration signals are processed using a high-pass filter and a low-pass filter with a cutoff frequency of 50Hz, respectively. The components with frequencies higher than 50Hz are the triaxial BCV signals, and the components with frequencies lower than 50Hz are the triaxial ECDM signals. In step 1.2, a linear BCV signal is introduced, which is the sum of the squares of the X, Y, and Z axis data of the triaxial BCV signal to the power of 1 / 2. A linear ECDM signal is also introduced, which is the sum of the squares of the X, Y, and Z axis data of the triaxial ECDM signal to the power of 1 / 2.

3. The speech content recognition method using an ear-worn motion sensor as described in claim 1, characterized in that, In step 1.2, the threshold T0 is set to [0.1, 0.5] times the maximum value of the linear BCV signal and the linear ECDM signal.

4. The speech content recognition method using an ear-worn motion sensor as described in claim 2, characterized in that, The threshold T0 is set to 0.2 times the maximum value of the linear BCV signal and the linear ECDM signal.

5. The speech content recognition method using an ear-worn motion sensor as described in claim 2, characterized in that, In step 2.1, the deep regression network consists of an input layer, two fully connected layers, and an output layer. The input layer processes the noise-affected triaxial ECDM signal, calculates the spectral density of its X, Y, and Z data using short-time Fourier transform, and inputs it to the fully connected layer. The fully connected layer eliminates motion noise, and the output layer uses inverse short-time Fourier transform to restore the noise-free triaxial spectral density to a clean triaxial ECDM signal. During the deep regression network training phase, acceleration signals during speech are collected in a noise-free environment using an ear-worn device, and acceleration signals during daily human activities are collected in a non-speaking environment. Step 1 yields a noise-free, clean triaxial ECDM signal and a triaxial body motion noise signal. A triaxial ECDM signal with body motion noise interference is artificially synthesized by linearly superimposing [0.1, 1] times the body motion noise onto the clean triaxial ECDM signal. The artificially synthesized noisy triaxial ECDM signal is used as the input signal to the network input layer, and the network parameters are optimized by minimizing the mean square error between the output signal and the corresponding clean triaxial ECDM signal. During the usage phase, the noisy triaxial ECDM signal is input into the trained deep regression neural network, and the output is the clean triaxial ECDM signal.

6. The speech content recognition method using an ear-worn motion sensor as described in claim 1, characterized in that, In step 2.2, the upper envelope E of the X, Y, and Z axis data of the triaxial ECDM signal is extracted respectively. x E y E z As a characteristic signal of EDCM, E = {E x E y E z In step 2.3, the amplitude of the ECDM characteristic signal is changed using a Procrustes transform-based method to eliminate the influence of individual differences. First, an alignment method based on dynamic time warping is used to stretch the X, Y, and Z axis data of the ECDM characteristic signals from multiple users to the same data length, and then the average signal is calculated. Where n is the number of users. This represents the average signal along the X-axis of the ECDM characteristic signals after stretching by n users; and Let Y and Z represent the average signals of the stretched ECDM feature signals from n users, respectively. A Procrustes transformation is performed on the current ECDM feature signal E to be converted, involving rotation, scaling, and translation. The result of the transformation is: Where H is the rotation matrix, α is the scaling factor, and β is the translation factor, with α and β being constants; for Perform singular value decomposition: U and V are orthogonal, Γ is a diagonal matrix, T denotes matrix transpose, and the rotation matrix H passes through H = UV. T Calculate; use the least squares method to search for reasonable values ​​of α and β, so that all transformed ECDM feature signals have the minimum mean square error.

7. The speech content recognition method using an ear-worn motion sensor as described in claim 1, characterized in that, In step 3.1, a clean BCV signal is extracted by eliminating noise caused by the vibration of the ear-worn device through pitch tracing. Pitch tracing includes signal decomposition, channel aggregation, pitch tracing, and signal synthesis. First, signal decomposition is performed. The filtered signal is analyzed using a sliding window, and the signal of the c-th filter channel in the w-th window is represented as e. c,w (t); then channel aggregation is performed, and the correlation coefficient S between adjacent channels is defined. c,w The following is how the signals e of adjacent sub-channels are quantized. w,c (t) and e w,c+1 Correlation of (t): in, A c,w (τ) represents the signal e w,c The autocorrelation coefficient of (t) when the time lag is τ. For the expectation operator, For signal e w,c The mean of (t), Let A be the autocorrelation coefficient. c,w The mean of (τ); compared with S c,w If the threshold T1 is exceeded, two channels are aggregated into one channel; after iteratively processing all channel signals within the window, several channels are obtained; then, taking advantage of the characteristic that each user's speech has a constant pitch over a short period of time, the signals within a sliding window with the same pitch are tracked; the c-th channel signal e within the sliding window w is calculated. w,c The fundamental period of (t), that is, let Time lag value for obtaining the maximum value Calculate the global pitch period of the original BCV signal within the sliding window w, i.e., let Time lag value for obtaining the maximum value Compare the fundamental period of each channel signal within the sliding window. With global pitch period If the difference is less than the threshold T2, the signal of this channel originates from the wearer's voice signal and is retained for synthesizing a pure BCV signal; finally, a pure BCV signal is synthesized; the signal in each sliding window is processed continuously, the voice signal originating from the wearer is separated according to the global pitch period, the separated signal is processed by a filter, and then all channels are summed to obtain a pure BCV signal.

8. The speech content recognition method using an ear-worn motion sensor as described in claim 7, characterized in that, In step 3.1, T1 is set to a value range of [0.5-0.99], and T2 is set to a value range of [0.6, 1].

9. A speech content recognition method using an ear-worn motion sensor as described in claim 8, characterized in that, T1 is set to 0.985, and T2 is set to 0.

985.

10. A speech content recognition method using an ear-worn motion sensor as described in claim 1, characterized in that, In step 4, a deep neural network is constructed based on CNN and CTC, including a feature signal processor, a feature fusionist, and a speech content recognizer. The feature signal processor is further divided into two sub-networks for processing. Each sub-network consists of multiple convolutional layers, max pooling layers, batch normalization (BN) layers, dropout layers, and flattening layers. The convolutional layers extract implicit speech content information, the max pooling layers perform subsampling and nonlinear compression, the BN layers further improve model performance and stability, the dropout layers prevent overfitting, and finally, the flattening layers combine the obtained information to generate a one-dimensional latent feature. The BCV sub-network takes the BCV feature signal as input; after processing by the BCV sub-model, it outputs a one-dimensional feature vector Ψ containing speech content information. BCV The ECDM subnetwork takes the ECDM feature signal as input and outputs a one-dimensional feature vector Ψ containing speech content information. ECDM The feature fusion unit combines the feature vectors Ψ output by the ECDM subnetwork and the BCV subnetwork in the feature signal processor. ECDM and Ψ BCV Bilinear pooling is used to fuse the features into a single one-dimensional feature; the bilinear pooling process involves calculating the outer product of the latent representations of the ECDM feature vector and the BCV feature vector. For the Kronecker integrator, The matrix vectorization operator is used; then, the L2 norm is applied to Ψ to obtain the final fused features; finally, the speech content recognizer classifies the fused features, using a softmax layer to obtain the probability that the current input is a certain word, and the word p with the highest probability is selected. t This is the recognition result; the CTC layer processes the softmax layer to accumulate the recognized sequence of repeated words y = y1, ..., y2. t ,…,y T To obtain the final identified word sequence y * Define CTCloss as: Loss CTC =-ln(P(y) * |y)) where P(*|*) represents the conditional probability operator; during the training phase of a deep neural network, y is known * The parameters of the deep learning model are updated iteratively using steps 1 to 4 to obtain y, and the loss is minimized. CTC The deep neural network constructed is trained.

Images