Mouth shape identification method and system based on ear canal sound wave reflection signal processing

Abstract

The invention belongs to the field of auditory meatus sound wave reflection signal processing, and discloses a mouth shape recognition method and system based on auditory meatus sound wave reflection signal processing, noise and interference components in signals are eliminated, and sound wave signals are subjected to preprocessing such as cutting and tailoring; obtaining an optimal regular path between the two time sequences; useful information is extracted; performing preprocessing and feature extraction on the signals; acoustic wave signal features are extracted through Fourier transform, frequency domain energy calculation, feature value calculation and the like; and carrying out KNN classifier classification. Theoretical and technical supports are provided for improving the precision and robustness of mouth shape recognition, exploring the application potential of ear canal sound wave reflection signal processing in mouth shape recognition and promoting development of a mouth shape recognition technology achieved through ear canal sound wave reflection signal processing in practical application.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the technical field of ear canal sound wave reflection signal processing, and in particular relates to a lip shape recognition method and system based on ear canal sound wave reflection signal processing. Background Art

[0002] With the rapid development of information technology, the human-computer interaction mode has gradually changed from the traditional keyboard and mouse to a more natural and intuitive mode. The widespread application of technologies such as speech recognition and gesture recognition has enabled people to interact with computers more conveniently. The existing human-computer interaction technologies are mainly as follows: 1. Voice interaction system. This system is based on speech recognition and synthesis technology, allowing users to interact with machines through natural language. Users issue instructions or inquiries in spoken form, and the system responds through voice feedback. However, this system faces the challenge of speech recognition accuracy, especially in noisy environments or when facing dialect accents, its recognition effect is greatly reduced. In addition, voice interaction also involves the complexity of natural language understanding. How to accurately understand the user's intentions and make appropriate responses is also a major problem that current technology needs to overcome. 2. Virtual reality (VR) and augmented reality (AR) interaction technology. These two technologies provide users with an immersive interactive experience by simulating or augmenting the real environment. Users can interact with machines in the virtual world and feel an unprecedented sense of immersion. However, the implementation of VR and AR technologies requires high-performance hardware support, and the equipment cost is relatively high. In addition, long-term use of these devices can cause discomfort or health risks to users, such as visual fatigue and dizziness. Therefore, how to reduce equipment costs and reduce user discomfort while ensuring immersion is a key issue that VR and AR technologies need to solve. These technologies still have certain limitations, so it is particularly important to explore a new and more accurate way of human-computer interaction.

[0003] As a new interactive method, lip recognition technology based on acoustic wave signals reflected from the ear canal has important research value and application prospects. In recent years, foreign countries have begun to pay close attention to and attach great importance to human-computer interfaces, which has become the focus of many scholars and has achieved many research results. The research team of the University of Tsukuba monitored the air pressure in the ear canal and matched it with facial-related movements, and then designed a jaw, face and head movement recognition system called CanalSense, which uses a barometer embedded in the earphones to identify facial-related movements. They found that facial-related movements changed the air pressure in the ear canal, and this change depended on the type and degree of movement. They also found that this feature change can be used to identify facial-related movements. And an experiment was conducted to measure the accuracy of recognition. The results showed that for 11 facial-related movements, the recognition accuracy of each user was 87.6%, and for 4 mouth-opening levels, the recognition accuracy of each user was 87.5%.

[0004] The research team at Graz University of Technology extracted simple temporal features from the in-ear pressure signal and designed a system called BARTON to identify tongue movements (left, right, and forward). It is a low-power and robust tongue movement sensing system based on a barometer, using a low sampling rate of less than 50Hz and extracting only simple temporal features from the in-ear pressure signal, proving that it is feasible to distinguish important tongue movements at low power consumption. And they designed a prototype for BARTON, which integrates commercial headphones for in-ear pressure sensing and an ARM microcontroller for signal processing. Evaluations show that BARTON has a classification accuracy of 94% and a power consumption of 8.4mW, achieving comparable accuracy, but with power consumption 44 times lower than the most advanced microphone-based solutions.

[0005] The University of Maryland research team designed a new contactless sensor called Tongue-n-Cheek that uses a micro-radar array to capture tongue movements. The micro-radar array acts as a proximity sensor and captures muscle movements as participants make tongue movements. Tongue-n-Cheek converts these movements into gestures using a novel signal processing algorithm. We demonstrated the efficacy of Tongue-n-Cheek and showed that our system can reliably infer gestures with 95% accuracy and low latency.

[0006] The research team at the University of Washington designed a non-invasive tongue-machine interface called TongueSee, which can recognize a rich set of tongue gestures using electromyography (EMG) signals from the skin surface. They demonstrated the feasibility and robustness of TongueSee through an experimental study of classifying six tongue gestures of eight participants. Using a three-protrusion preamble design, Tongue-See achieved a classification accuracy of 94.17% and a false alarm probability of 0.000358 per second.

[0007] In recent years, China has also attached great importance to and paid attention to human-computer interfaces, and many scholars have conducted in-depth research on non-invasive human-computer interfaces. The Beijing Institute of Technology team designed a non-invasive tongue-jaw movement recognition system called CanalScan, which uses a speaker and a microphone to send sound signals, then captures its reflection and obtains the unique pattern of ear canal deformation caused by tongue-jaw movement. The dynamic segmentation method described in the support vector domain is used to segment tongue-jaw movement. In order to overcome the sensor position sensitivity defects and ear canal shape sensitivity defects in multipath reflection, they first designed an algorithm to help users adjust the acoustic sensor to the same effective area. Then, they proposed a data conversion mechanism to reduce the impact of differences in ear canal shape and relative position between the sensor and the ear canal. CanalScan explores 12 unique and consistent features and applies a random forest classifier to distinguish tongue-jaw movements. Extensive experiments on 20 participants showed that CanalScan achieves reliable recognition of 6 tongue-jaw movements, is robust to various usage scenarios, and can be extended to new users without retraining and adaptation.

[0008] The Shandong University team has developed an interactive technology called Tongible that uses the tongue as input without any mechanical or electromagnetic assistive devices. In Tongible, an RGB camera captures six tongue movements and uses them as basic control actions. In recent years, people have studied the use of the tongue for people with no or minimal upper limb function to use computers. These studies have mainly focused on the use of mechanical or electromagnetic devices. However, these devices must come into contact with people's mouths and can lead to hygiene issues or accidental ingestion. This work presents preliminary usability tests showing that Tongible is effective for people with dexterity impairments in pointing and text input.

[0009] In view of the above analysis, the technical problems that need to be solved urgently in the prior art are:

[0010] Research on human-computer interfaces has been very in-depth both at home and abroad. A large number of studies have shown that signals such as air pressure in the ear canal, reflected sound waves, and facial muscle currents are related to the movements of the tongue, upper and lower jaws, face, and head. However, there is a lack of in-depth analysis and research on lip shape changes and sound wave reflection signals in the ear canal. Summary of the invention

[0011] In view of the problems existing in the prior art, the present invention provides a lip shape recognition method and system based on processing of ear canal sound wave reflection signals.

[0012] The present invention is implemented as follows: a lip shape recognition method based on processing of ear canal sound wave reflection signals, characterized in that the lip shape recognition method based on processing of ear canal sound wave reflection signals specifically comprises:

[0013] S1: Signal preprocessing, eliminating noise and interference components in the signal, and cutting and trimming the sound wave signal;

[0014] S2: Dynamic time warping, obtaining the best warping path between two time series;

[0015] S3: Convolution transformation, extracting useful information;

[0016] S4: wavelet transform, signal preprocessing and feature extraction;

[0017] S5: Feature extraction, using Fourier transform, frequency domain energy calculation and eigenvalue calculation to extract the characteristics of the acoustic wave signal;

[0018] S6: Classification by KNN classifier.

[0019] Furthermore, the S1, preprocessing operation includes steps such as filtering, noise reduction, and normalization, and the collected sound wave signals are cut, cropped, and other preprocessing is performed to distinguish the lip movement stage and the static stage.

[0020] Furthermore, the implementation of the dynamic time warping algorithm in S2 mainly includes the following steps:

[0021] (1) Initializing the regularized path: When the algorithm starts, it is necessary to determine the starting point and end point of the regularized path. Usually, the starting point of the regularized path is the first point of the two time series, and the end point is the last point;

[0022] (2) Calculate the cumulative distance: Starting from the starting point of the regularized path, gradually calculate the distance between the corresponding points of the two time series and add it to the cumulative distance of the current position. The distance calculation here can use different measurement methods such as Euclidean distance and Manhattan distance;

[0023] (3) Finding the best path: At each position, the algorithm needs to select the next moving direction, including upward, rightward, and diagonally to the upper right. The algorithm compares the cumulative distances in different directions and selects the direction with the minimum cumulative distance as the next moving direction.

[0024] (4) Update the regular path: Update the current position of the regular path according to the selected moving direction, and continue to calculate the cumulative distance of the next position;

[0025] (5) Termination condition: When the regularized path reaches the end point, the algorithm terminates. At this time, all points on the regularized path represent the optimal correspondence between the two time series.

[0026] Further, the algorithm steps of S3, the convolution transformation of the acoustic wave signal, mainly include the following aspects:

[0027] (1) Signal preprocessing: preprocessing the input sound wave signal, including denoising, filtering and other operations;

[0028] (2) Convolution kernel design: The convolution kernel is designed according to the application requirements. The convolution kernel can be fixed or learnable. When designing the convolution kernel, factors such as its shape, size, and parameters need to be considered.

[0029] (3) Convolution operation: The designed convolution kernel is convolved with the input signal through a sliding window method, that is, the convolution kernel is slid on the input signal and the convolution result at each position is calculated.

[0030] Further, the S4, wavelet transform algorithm mainly includes the following steps:

[0031] (1) Select mother wavelet: Commonly used mother wavelets include Haar wavelet, Daubechies wavelet, Morlet wavelet, etc. When selecting a mother wavelet, it is necessary to consider the characteristics of the signal and the purpose of analysis;

[0032] (2) Determine the number of decomposition layers: When determining the number of decomposition layers, a trade-off needs to be made between resolution and computational complexity;

[0033] (3) Perform wavelet decomposition on the acoustic signal: perform inner product operations on the acoustic signal and wavelet functions of different scales to obtain wavelet coefficients;

[0034] (4) Processing the wavelet coefficients: performing threshold processing, reconstruction and other operations on the wavelet coefficients;

[0035] (5) Result presentation and analysis: The processed sound wave frequency domain signal is visualized. By comparing the frequency domain images of the sound wave signal reflected from the ear canal at different mouth shapes after wavelet transformation, a preliminary analysis is conducted on the correlation between the mouth shape change and the sound wave reflected from the ear canal.

[0036] Furthermore, the algorithm flow of the acoustic signal feature extraction in S5 mainly includes:

[0037] (1) Signal preprocessing: including noise removal, filtering, sampling rate adjustment, etc., to improve signal quality and stability and eliminate interference components in the signal;

[0038] (2) Fourier transform: Use the fast Fourier transform (FFT) algorithm to convert it from the time domain to the frequency domain to obtain the amplitude and phase information of the sound wave signal in the frequency domain;

[0039] (3) Frequency domain energy calculation: The energy of the sound wave signal is obtained by calculating the square of the amplitude or power of each frequency component;

[0040] (4) Eigenvalue calculation: The eigenvalue selected in this experiment is the average value of the frequency domain energy and arithmetic power of the acoustic wave signal.

[0041] Further, the S6 uses a KNN classifier for classification, and the steps are as follows:

[0042] (1) Calculate the distance: For a new sample to be classified, calculate the distance between it and each sample in the training set;

[0043] (2) Select K nearest neighbors: Based on the calculated distance, select the K samples closest to the new sample as the nearest neighbors;

[0044] (3) Determine the category: Based on the category labels of these K nearest neighbors, the category of the new sample is determined by majority voting or other methods. If there are multiple different category labels among the K nearest neighbors, the category with the most occurrences is usually selected as the prediction result.

[0045] Another object of the present invention is to provide a lip recognition system based on processing of ear canal sound wave reflection signals, the system specifically comprising:

[0046] Input module, used to acquire signals and perform preprocessing;

[0047] Dynamic time warping module, used for dynamic time warping of acoustic wave signals;

[0048] Feature extraction module, which extracts the features of acoustic wave signals through convolution transform, wavelet transform, Fourier transform, etc.;

[0049] Classification module, uses KNN classifier for classification.

[0050] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0051] First, the present invention proposes a lip shape recognition technology based on processing of ear canal sound wave reflection signals, which provides theoretical and technical support for improving the accuracy and robustness of lip shape recognition, exploring the application potential of ear canal sound wave reflection signal processing in lip shape recognition, and promoting the development of lip shape recognition technology based on processing of ear canal sound wave reflection signals in practical applications.

[0052] Second, the expected benefits and commercial value after the transformation of the technical solution of the present invention are: This technology hopes to establish a relatively complete lip recognition system through in-depth research by processing the ear canal sound wave reflection signal, providing new technical support for the fields of intelligent interaction, assisted communication for the deaf and mute, etc.

[0053] The technical solution of the present invention fills the gap in the technology in the industry at home and abroad: the technology can accurately recognize the changes in lip shape by capturing and analyzing the sound wave signals reflected in the ear canal. Compared with traditional speech recognition and gesture recognition technologies, this technology has higher sensitivity and accuracy, especially in noisy environments or in the recognition of fine movements, and has unique advantages.

[0054] Third, the present invention solves the shortcomings of the lip shape recognition method in the prior art in terms of accuracy and practicality. Traditional lip shape recognition technology mostly relies on visual data obtained by the camera, but is limited by factors such as ambient light, shooting angle, and facial occlusion, resulting in reduced recognition accuracy or unusability. In contrast, the present invention realizes lip shape recognition through ear canal sound wave reflection signal processing, which does not rely on external environmental conditions, greatly improving the reliability and adaptability of recognition.

[0055] The present invention realizes accurate extraction and analysis of useful information in sound wave signals through multiple steps such as signal preprocessing, dynamic time warping, convolution transform, wavelet transform and feature extraction. In particular, dynamic time warping solves the problem of different people's speaking speed differences, and combines wavelet transform and Fourier transform to accurately capture the sound wave reflection characteristics caused by mouth shape changes, thereby ensuring high accuracy in the recognition process. This multi-step optimization process enables the system to more accurately identify different mouth shape patterns, overcoming the problems of information loss and noise interference in traditional recognition methods.

[0056] In industrial applications, the significant technical progress of the present invention is reflected in its ability to be widely applicable to adverse use environments, such as low light and scenes with large noise interference, and it can still maintain efficient recognition performance. This method does not rely on a complex camera system, but only needs to collect the sound wave reflection signal in the ear canal to complete high-precision lip recognition. Its application range is wider, not only can it be used in human-computer interaction devices, but also in occasions that require privacy protection, such as non-visual perception interaction technology in hearing aids and smart headphones.

[0057] In addition, the lip recognition technology of the present invention can play an important role in the medical field, especially in auxiliary devices for the hearing-impaired, helping them to achieve more natural voice control and human-computer interaction through lip recognition. At the same time, the low power consumption design and high accuracy of this technology make it have high commercial value and market potential, which significantly promotes the practical application of lip recognition technology in various intelligent devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a lip shape recognition method based on processing of ear canal sound wave reflection signals provided by an embodiment of the present invention;

[0059] Figure 2It is a frequency domain waveform diagram of each group after the convolution operation provided by the embodiment of the present invention;

[0060] Figure 3 It is a frequency domain waveform diagram after convolution and wavelet transformation of the sound wave frequency domain provided by an embodiment of the present invention;

[0061] Figure 4 is a module diagram of a lip shape recognition system based on processing of ear canal sound wave reflection signals provided by an embodiment of the present invention;

[0062] Figure 5 It is a frequency domain waveform diagram after convolution and wavelet transformation of the sound wave frequency domain provided by an embodiment of the present invention;

[0063] Figure 6 It is a schematic diagram of the features attempted to be selected and whether they have distinguishing properties provided by an embodiment of the present invention;

[0064] Figure 7 : is a distribution diagram of characteristic values ​​(frequency domain energy) of three mouth-shaped sound wave reflection signal data sets provided by an embodiment of the present invention, wherein the abscissa is the xth experiment and the ordinate is the corresponding sound wave frequency domain energy;

[0065] Figure 8 : is a distribution diagram of characteristic values ​​(arithmetic mean power) of three mouth-shaped sound wave reflection signal data sets provided by an embodiment of the present invention, wherein the abscissa is the xth experiment and the ordinate is the corresponding arithmetic mean power;

[0066] Fig. 9 It is a ratio diagram of test prediction and actual results of the test set data provided by the embodiment of the present invention (using frequency domain energy as the eigenvalue);

[0067] Fig.10 It is a ratio diagram of test prediction and actual results of the test set data provided by the embodiment of the present invention (using arithmetic mean power as the characteristic value);

[0068] Fig.11 It is a ratio diagram of test prediction and actual results of the test set data provided by an embodiment of the present invention (using frequency domain energy and arithmetic mean power for feature combination as feature values). DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0070] The working principle of the lip shape recognition method provided by the present invention can be divided into four key steps:

[0071] 1. Signal preprocessing and noise elimination

[0072] In the first step, the system preprocesses the collected ear canal sound wave reflection signal, mainly to eliminate noise and interference components in the signal. The preprocessing process includes cutting and trimming the signal to ensure that the useful signal related to lip recognition is retained, while removing irrelevant information to provide purer data input for subsequent analysis.

[0073] 2. Dynamic Time Warping and Convolution Transform

[0074] Next, the system uses the Dynamic Time Warping (DTW) method to compare the two time series and find the best alignment path between them. This step is mainly used to solve the signal misalignment problem caused by the difference in the speed of speech of different people. Subsequently, through the convolution transformation, the useful information in the signal is further extracted, and the complex spatiotemporal features are compressed into high-dimensional features that can be used for analysis.

[0075] 3. Wavelet transform and feature extraction

[0076] After completing basic signal processing, the system will perform wavelet transform to decompose the signal into different frequency bands to capture the changes in sound wave reflection caused by changes in lip shape in a more fine-grained manner. Then, Fourier transform, frequency domain energy calculation, and eigenvalue calculation are used to further extract frequency and time domain features in the sound wave signal. These features will serve as the basis for classification and represent different lip shape patterns.

[0077] 4. Application of KNN Classifier

[0078] Finally, the system inputs the extracted features into the K nearest neighbor (KNN) classifier for pattern classification. The KNN classifier determines the mouth shape corresponding to the current sound wave signal by calculating the similarity between the new input signal and the signal in the training set. This classification process can distinguish different mouth shapes based on the difference in feature values ​​and achieve the goal of mouth shape recognition.

[0079] The following are two specific embodiments of the lip shape recognition method based on processing the ear canal sound wave reflection signal:

[0080] Example 1: Lip recognition in a voice control system

[0081] Voice control systems are widely used in smart homes, car devices and smart assistants. However, in noisy environments, voice recognition alone is easily interfered by background noise. By combining lip recognition, the robustness and accuracy of voice control systems can be enhanced, especially in scenarios with high ambient noise.

[0082] This method is used to assist speech recognition by collecting sound wave reflection signals in the user's ear canal:

[0083] 1) Signal acquisition and preprocessing: When the user wears headphones, the built-in microphone collects the sound wave reflection signal in the ear canal. The signal is noise-cancelled and dynamically time-normalized to ensure signal purity and time alignment.

[0084] 2) Feature extraction and classification: After convolution and wavelet transform processing, the spectral features and energy features of the signal are extracted. These features are input into the KNN classifier and matched with the pre-stored lip shape features.

[0085] 3) Application of results: The recognized lip shape information is combined with the speech recognition results to further improve the accuracy of voice command recognition. In a noisy environment, the system can recognize the user's intention through lip shape information, thereby improving the response accuracy of smart devices.

[0086] Through this embodiment, the voice control system can effectively cope with the interference of environmental noise, significantly improve the recognition accuracy of the system, and enhance the user experience.

[0087] Example 2: Pronunciation Correction System in Language Learning

[0088] In language learning, especially in the process of learning a second language, accurate matching of mouth shape and pronunciation is crucial. Traditional speech recognition systems cannot accurately determine whether the learner's mouth shape matches the correct pronunciation. Through this method, dual detection of learners' mouth shape and pronunciation can be achieved, helping them to better master pronunciation skills.

[0089] In the pronunciation correction system, the ear canal sound wave reflection signal processing is used to realize lip shape recognition and assist pronunciation training:

[0090] 1) Signal acquisition and analysis: When learners are practicing pronunciation, the system collects the sound wave reflection signals in the ear canal through the sensor in the earphones. After eliminating the background noise, it performs dynamic time warping and convolution transformation to extract effective pronunciation features.

[0091] 2) Matching of mouth shape and pronunciation features: The system performs Fourier transform and wavelet transform on the extracted sound wave features to obtain the feature values ​​corresponding to the mouth shape. Through the KNN classifier, the learner's actual mouth shape is matched with the mouth shape database of correct pronunciation.

[0092] 3) Feedback and correction: The system provides real-time feedback on whether the mouth shape matches the correct pronunciation, and provides improvement suggestions, such as "open your mouth wider" and "raise tongue position", to help learners adjust their mouth shape to pronounce the correct syllables.

[0093] Through this embodiment, learners can get real-time pronunciation correction guidance, correct incorrect pronunciation habits, and accelerate the process of language learning. The system can effectively identify the differences between learners' mouth shapes and pronunciations, and provide targeted improvement suggestions, greatly improving the effect of pronunciation training.

[0094] like Figure 1 As shown, an embodiment of the present invention provides a lip shape recognition method based on processing of ear canal sound wave reflection signals, and the method specifically includes:

[0095] S1: Signal preprocessing, eliminating noise and interference components in the signal, and cutting and trimming the sound wave signal;

[0096] S2: Dynamic time warping, obtaining the best warping path between two time series;

[0097] S3: Convolution transformation, extracting useful information;

[0098] S4: wavelet transform, signal preprocessing and feature extraction;

[0099] S5: Feature extraction, using Fourier transform, frequency domain energy calculation and eigenvalue calculation to extract the characteristics of the acoustic wave signal;

[0100] S6: Classification by KNN classifier.

[0101] 1) Signal preprocessing and dynamic time warping: In the lip recognition method, the sound wave reflection signal collected from the ear canal is first preprocessed through step S1. This stage includes operations such as noise elimination, interference component removal, signal cutting and clipping to ensure the purity and consistency of the input signal. The preprocessed signal is input to step S2 for dynamic time warping (DTW), which is a process used to calculate the optimal warping path between two time series so that signal sequences at different times can be effectively aligned, laying the foundation for subsequent signal analysis and feature extraction.

[0102] 2) Signal analysis of convolution transform and wavelet transform: In step S3, the regularized signal is processed by convolution transform, which can extract useful information from the signal and filter out irrelevant noise and interference signals. This stage ensures that the subsequent steps can be analyzed based on high-quality signals. In step S4, the wavelet transform is applied to further preprocess and extract features of the signal. Wavelet transform has good time-frequency localization characteristics, which can effectively decompose the signal and extract feature information at different scales, providing rich feature data for the recognition process.

[0103] 3) Fourier transform and feature extraction: In step S5, feature extraction is performed on the processed signal. First, the time domain signal is converted into a frequency domain signal using Fourier transform, from which frequency domain features, such as spectrum features and harmonic components, are extracted. At the same time, frequency domain energy calculation is performed to determine the energy distribution of the signal in different frequency bands. Through these operations, characteristic values ​​representing the key features of the sound wave signal can be obtained, providing accurate data support for the subsequent classification steps.

[0104] 4) Lip shape recognition by KNN classifier: Finally, in step S6, the extracted features are classified using the K nearest neighbor classifier (KNN). The KNN algorithm determines the lip shape category to which the input signal belongs based on the similarity between the input features and the samples in the training data set. Through this process, the ear canal sound wave reflection signal can be accurately identified as a specific lip shape category, realizing lip shape recognition based on the sound wave reflection signal. This method combines multiple signal processing and feature extraction techniques to ensure the accuracy and robustness of the recognition results.

[0105] The S1, preprocessing operation includes filtering, noise reduction, normalization and other steps, aiming to eliminate noise and interference components in the signal, making the signal clearer and easier to analyze. Secondly, the collected sound wave signal is cut, cropped and other preprocessing to distinguish the lip movement stage and the static stage.

[0106] The implementation of the S2, dynamic time warping algorithm mainly includes the following steps:

[0107] (1) Initialize the regularized path: When the algorithm starts, it is necessary to determine the starting point and end point of the regularized path. Usually, the starting point of the regularized path is the first point of the two time series, and the end point is the last point.

[0108] (2) Calculate the cumulative distance: Starting from the starting point of the regularized path, gradually calculate the distance between the corresponding points of the two time series and add it to the cumulative distance of the current position. The distance calculation here can use different metrics such as Euclidean distance and Manhattan distance.

[0109] (3) Finding the best path: At each position, the algorithm needs to choose the next moving direction, including upward, right, and diagonally to the upper right. The algorithm compares the cumulative distances in different directions and selects the direction with the smallest cumulative distance as the next moving direction.

[0110] (4) Update the regular path: Update the current position of the regular path according to the selected moving direction, and continue to calculate the cumulative distance of the next position.

[0111] (5) Termination condition: When the regularized path reaches the end point, the algorithm terminates. At this point, all points on the regularized path represent the best correspondence between the two time series.

[0112] Through the above steps, the DTW algorithm can obtain the best warping path between two time series, thereby realizing dynamic time warping of acoustic wave signals. This warping method not only takes into account the temporal changes of the signal, but also retains the main features of the signal, thus improving the accuracy of signal matching.

[0113] The algorithm steps of S3, the convolution transformation of the acoustic signal, mainly include the following aspects:

[0114] (1) Signal preprocessing: Preprocess the input sound wave signal, including denoising, filtering and other operations to improve the signal quality.

[0115] (2) Convolution kernel design: Design the convolution kernel according to the application requirements. The convolution kernel can be fixed or learnable. When designing the convolution kernel, factors such as its shape, size, and parameters need to be considered.

[0116] (3) Convolution operation: The designed convolution kernel is convolved with the input signal. This is achieved by sliding the convolution kernel over the input signal and calculating the convolution result at each position.

[0117] Due to the complexity of the ear canal structure and the diversity of sound wave propagation, the reflected sound wave signals often have problems such as noise interference and frequency aliasing. Therefore, we need to use effective signal processing methods to extract useful information. In this process, convolution of the blank control group and the experimental group frequency domain has become an effective means. The frequency domain waveforms of each group after convolution operation are as follows: Figure 2 shown.

[0118] The blank control group is the sound wave signal reflected from the ear canal when no lip shape changes were made, while the experimental group is the signal when three lip shape changes were made. By convolving these two sets of signals in the frequency domain, we can observe their mutual influence and interaction in frequency characteristics, thereby revealing the characteristics related to lip shape changes.

[0119] Specifically, the frequency domain convolution operation can simulate the superposition and modulation process of the signal in frequency. When we convolve the frequency domain signals of the blank control group and the experimental group, the specific frequency components in the experimental group signal will be multiplied and superimposed with the frequency components in the blank control group signal. This process will highlight the frequency features related to the changes in lip shape, making it easier to identify the part of the experimental group signal related to the lip shape.

[0120] In addition, frequency domain convolution can also help us eliminate noise interference and frequency aliasing problems. By convolving the blank control group and the experimental group signals, we can separate the noise components from the useful signals, thereby extracting purer lip shape features. At the same time, the convolution operation can also rearrange and combine the frequency components, so that the originally aliased frequency components can be separated and identified.

[0121] The S4, wavelet transform algorithm mainly includes the following steps:

[0122] (1) Selecting mother wavelet: The selection of mother wavelet has an important influence on the result of wavelet transform. Commonly used mother wavelets include Haar wavelet, Daubechies wavelet, Morlet wavelet, etc. When selecting mother wavelet, it is necessary to consider the characteristics of the signal and the purpose of analysis.

[0123] (2) Determine the number of decomposition layers: The number of decomposition layers determines the fineness of the wavelet transform. The more layers there are, the higher the frequency resolution will be, but the computational complexity will also increase accordingly. Therefore, when determining the number of decomposition layers, it is necessary to strike a balance between resolution and computational complexity.

[0124] (3) Perform wavelet decomposition on the acoustic signal: Perform inner product operations on the acoustic signal and wavelet functions of different scales to obtain wavelet coefficients. These coefficients reflect the characteristics of the signal at different time and frequency scales.

[0125] (4) Processing the wavelet coefficients: performing threshold processing, reconstruction and other operations on the wavelet coefficients.

[0126] (5) Results display and analysis: The processed sound wave frequency domain signal is visualized to facilitate intuitive analysis of the time-frequency characteristics of the sound wave signal. By comparing the frequency domain images of the sound wave signal reflected from the ear canal at different mouth shapes after wavelet transformation, the correlation between the mouth shape change and the sound wave reflected from the ear canal can be preliminarily analyzed.

[0127] The frequency domain waveform after convolution and wavelet transform of the sound wave frequency domain is as follows Figure 3 As shown, according to the running results, we found that after the reflected sound wave signal was processed by the above process, the frequency domain waveform diagram showed similarities between the sound waves of the same mouth shape and differences between the sound waves of different mouth shapes.

[0128] The algorithm flow of S5, acoustic signal feature extraction, mainly includes:

[0129] (1) Signal preprocessing: Before performing frequency domain energy analysis, the acoustic wave signal needs to be preprocessed. The preprocessing steps include noise removal, filtering, sampling rate adjustment, etc. to improve the quality and stability of the signal. Through preprocessing, the interference components in the signal can be eliminated, making the subsequent analysis more accurate and reliable.

[0130] (2) Fourier transform: The preprocessed sound wave signal needs to be Fourier transformed to convert it from the time domain to the frequency domain. The Fourier transform can be implemented using the fast Fourier transform (FFT) algorithm, which is efficient and accurate. Through the Fourier transform, we can obtain the amplitude and phase information of the sound wave signal in the frequency domain.

[0131] (3) Frequency domain energy calculation: In the frequency domain, the energy of the sound wave signal can be obtained by calculating the square of the amplitude or power of each frequency component. The specific calculation method can be determined according to the application requirements. Usually, we can calculate the square of the amplitude of each frequency component and add them up to get the frequency domain energy of the entire signal. In addition, we can also calculate the power of each frequency component to obtain the power spectrum of the signal.

[0132] (4) Eigenvalue calculation: Extracting useful characteristic information from frequency domain energy is a key step in acoustic signal analysis. The eigenvalues ​​selected in this experiment are the frequency domain energy and the arithmetic power average of the acoustic signal.

[0133] In the S6, in the lip shape recognition task based on the processing of ear canal sound wave reflection signal, the KNN (K-NearestNeighbors) classifier can be used as an effective classification method. By extracting the energy features of the sound wave frequency domain signal and using these features as input to the KNN classifier, accurate recognition of different lip shapes can be achieved. In practical applications, it is also necessary to select appropriate distance measurement methods, K values, and feature preprocessing steps according to specific data sets and task requirements to obtain the best classification performance.

[0134] The basic steps of KNN classifier are as follows:

[0135] (1) Calculate distance: For a new sample to be classified, the KNN algorithm calculates the distance between it and each sample in the training set.

[0136] (2) Select K nearest neighbors: Based on the calculated distance, select the K samples closest to the new sample as neighbors.

[0137] (3) Determine the category: Based on the category labels of the K nearest neighbors, determine the category of the new sample by majority voting or other methods. If there are multiple different category labels among the K nearest neighbors, the category with the most occurrences is usually selected as the prediction result.

[0138] like Figure 4 As shown, an embodiment of the present invention provides a lip recognition system based on processing of ear canal sound wave reflection signals, which specifically includes:

[0139] Input module, used to acquire signals and perform preprocessing;

[0140] Dynamic time warping module, used for dynamic time warping of acoustic wave signals;

[0141] Feature extraction module, which extracts the features of acoustic wave signals through convolution transform, wavelet transform, Fourier transform, etc.;

[0142] Classification module, uses KNN classifier for classification.

[0143] according to Figure 5 From the running results, we found that after the reflected sound wave signal was processed by the above process, the frequency domain waveform had similarities between the sound waves with the same mouth shape and differences between the sound waves with different mouth shapes.

[0144] We selected the more representative features in the sound wave frequency domain for feature extraction and classification experiments. Figure 6 As for whether the selected features are discriminative, we found that only frequency domain energy and arithmetic mean power are discriminative. Therefore, we finally selected these two as features for extraction and classification.

[0145] according to Figure 7 It can be seen that the frequency domain energy of the sound wave signal reflected by the ear canal has similarities between the sound waves of the same mouth shape and differences between the sound waves of different mouth shapes. Therefore, it can be used as a feature for classification.

[0146] according to Figure 8 It can be seen that the arithmetic mean power of the sound wave signal reflected by the ear canal also has similarities between the sound waves of the same mouth shape and differences between the sound waves of different mouth shapes. Therefore, it can be used as a feature for classification.

[0147] according to Fig. 9 It can be seen that mouth shapes one and two are easier to confuse, and the classification result of mouth shape three is more stable.

[0148] according to Fig.10 It can be seen that lip shape one and two are easier to confuse, the classification result of lip shape three is more stable, and the classification effect of using arithmetic mean power as the eigenvalue is better than using frequency domain energy as the eigenvalue.

[0149] according to Fig.11 It can be seen that lip shape one and two are easier to confuse, the classification result of lip shape three is more stable, and the classification effect of using frequency domain energy and arithmetic mean power as feature value is the same as the classification effect of using arithmetic mean power as feature value.

[0150] Validation: By observing and analyzing the processed signals, we can draw the following conclusions: (1) The frequency domain waveforms of the ear canal reflected sound wave signals processed by the above process in two experiments with different participants or the same participant with the same lip shape (i.e., in each column) are similar; (2) The frequency domain waveforms of the ear canal reflected sound wave signals processed with different lip shapes (i.e., in different columns) are different; (3) The recognition accuracy of this technology for lip shape three reaches 100%, and for lip shape one and lip shape two, it reaches 90% and 70% respectively.

[0151] Compared with traditional vision-based lip recognition, lip recognition technology based on sound wave reflection signals has the advantage of not being affected by environmental factors such as light and angle.

[0152] During the experiment, Figure 3 This is the image after convolution transform and wavelet transform of the sound waves reflected by the ear canal. From the geometric features of the image, it can be inferred that there are similarities in the characteristics of the sound waves reflected by the ear canal when different volunteers make the same lip shape, and there are obvious differences in the characteristics of the sound waves when the same volunteer makes different lip shapes.

[0153] In the final test results, Figure 7 The test set data provided by the embodiment of the present invention uses frequency domain energy as a characteristic value to test the distribution diagram. From the image, it can be seen that there are obvious differences in the test results when there are three mouth shapes, among which mouth shape 1 is more similar to mouth shape 2, and mouth shape 3 is more different from the other two mouth shapes;

[0154] Figure 8 The test set data provided by the embodiment of the present invention uses the arithmetic mean power as the characteristic value to test the distribution diagram. From the image, it can be seen that there are obvious differences in the test results when there are three mouth shapes, among which mouth shape one is more similar to mouth shape two, and mouth shape three is more different from the other two mouth shapes;

[0155] Fig. 9 The test set data provided by the embodiment of the present invention uses frequency domain energy as the feature value to perform test prediction and actual result ratio diagram. From the image, it can be seen that mouth shape one and two are more easily confused, and the classification result of mouth shape three is more stable;

[0156] Fig.10 The test set data provided by the embodiment of the present invention uses the arithmetic mean power as the feature value to perform the test prediction and the actual result ratio diagram. From the image, it can be seen that the mouth shape one and two are more easily confused, and the classification result of the mouth shape three is more stable;

[0157] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0158] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A lip shape recognition method based on processing of ear canal sound wave reflection signals, characterized in that: The method specifically includes: S1: Signal preprocessing, eliminating noise and interference components in the signal, and cutting and trimming the sound wave signal; S2: Dynamic time warping, obtaining the best warping path between two time series; S3: Convolution transformation, extracting useful information; S4: wavelet transform, signal preprocessing and feature extraction; S5: Feature extraction, using Fourier transform, frequency domain energy calculation and eigenvalue calculation to extract the characteristics of the acoustic wave signal; S6: Classification by KNN classifier.

2. The lip shape recognition method based on processing of ear canal sound wave reflection signals as claimed in claim 1, characterized in that: The S1, preprocessing operation includes steps such as filtering, noise reduction, and normalization, and performs preprocessing such as cutting and clipping on the collected sound wave signal to distinguish the lip movement stage and the static stage.

3. The lip shape recognition method based on processing of ear canal sound wave reflection signals as claimed in claim 1, characterized in that: The implementation of the S2, dynamic time warping algorithm mainly includes the following steps: (1) Initializing the regularized path: When the algorithm starts, it is necessary to determine the starting point and end point of the regularized path. Usually, the starting point of the regularized path is the first point of the two time series, and the end point is the last point; (2) Calculate the cumulative distance: Starting from the starting point of the regularized path, gradually calculate the distance between the corresponding points of the two time series and add it to the cumulative distance of the current position. The distance calculation here can use different measurement methods such as Euclidean distance and Manhattan distance; (3) Finding the best path: At each position, the algorithm needs to choose the next moving direction, including upward, right, and diagonally to the upper right. The algorithm compares the cumulative distances in different directions and selects the direction with the smallest cumulative distance as the next moving direction. (4) Update the regular path: Update the current position of the regular path according to the selected moving direction, and continue to calculate the cumulative distance of the next position; (5) Termination condition: When the regularized path reaches the end point, the algorithm terminates. At this time, all points on the regularized path represent the optimal correspondence between the two time series.

4. The lip shape recognition method based on processing of ear canal sound wave reflection signals as claimed in claim 1, characterized in that: The algorithm steps of S3, the convolution transformation of the acoustic signal, mainly include the following aspects: (1) Signal preprocessing: preprocessing the input sound wave signal, including denoising, filtering and other operations; (2) Convolution kernel design: The convolution kernel is designed according to the application requirements. The convolution kernel can be fixed or learnable. When designing the convolution kernel, factors such as its shape, size, and parameters need to be considered. (3) Convolution operation: The designed convolution kernel is convolved with the input signal through a sliding window method, that is, the convolution kernel is slid on the input signal and the convolution result at each position is calculated.

5. The lip shape recognition method based on processing of ear canal sound wave reflection signals as claimed in claim 1, characterized in that: The S4, wavelet transform algorithm mainly includes the following steps: (1) Select mother wavelet: Commonly used mother wavelets include Haar wavelet, Daubechies wavelet, Morlet wavelet, etc. When selecting a mother wavelet, it is necessary to consider the characteristics of the signal and the purpose of analysis; (2) Determine the number of decomposition layers: When determining the number of decomposition layers, a trade-off needs to be made between resolution and computational complexity; (3) Perform wavelet decomposition on the acoustic signal: perform inner product operations on the acoustic signal and wavelet functions of different scales to obtain wavelet coefficients; (4) Processing the wavelet coefficients: performing threshold processing, reconstruction and other operations on the wavelet coefficients; (5) Result presentation and analysis: The processed sound wave frequency domain signal is visualized. By comparing the frequency domain images of the sound wave signal reflected from the ear canal at different mouth shapes after wavelet transformation, a preliminary analysis is conducted on the correlation between the mouth shape change and the sound wave reflected from the ear canal.

6. The lip shape recognition method based on processing of ear canal sound wave reflection signals as claimed in claim 1, characterized in that: The algorithm flow of S5, acoustic signal feature extraction, mainly includes: (1) Signal preprocessing: including noise removal, filtering, sampling rate adjustment, etc., to improve signal quality and stability and eliminate interference components in the signal; (2) Fourier transform: Use the fast Fourier transform (FFT) algorithm to convert it from the time domain to the frequency domain to obtain the amplitude and phase information of the sound wave signal in the frequency domain; (3) Frequency domain energy calculation: The energy of the sound wave signal is obtained by calculating the square of the amplitude or power of each frequency component; (4) Eigenvalue calculation: The eigenvalue selected in this experiment is the average value of the frequency domain energy and arithmetic power of the acoustic wave signal.

7. The lip shape recognition method based on processing of ear canal sound wave reflection signals as claimed in claim 1, characterized in that: S6, using KNN classifier for classification, the steps are as follows: (1) Calculate the distance: For a new sample to be classified, calculate the distance between it and each sample in the training set; (2) Select K nearest neighbors: Based on the calculated distance, select the K samples closest to the new sample as the nearest neighbors; (3) Determine the category: Based on the category labels of these K nearest neighbors, the category of the new sample is determined by majority voting or other methods. If there are multiple different category labels among the K nearest neighbors, the category with the most occurrences is usually selected as the prediction result.

8. A lip recognition system based on processing of ear canal sound wave reflection signals as described in claims 1-7, characterized in that: The system specifically includes: Input module, used to acquire signals and perform preprocessing; Dynamic time warping module, used for dynamic time warping of acoustic wave signals; Feature extraction module, which extracts the features of acoustic wave signals through convolution transform, wavelet transform, Fourier transform, etc.; Classification module, uses KNN classifier for classification.

Images