A method and device for language recognition based on human mouth movement

Abstract

The present invention discloses a speech recognition method and device based on human mouth movement, belonging to the field of speech recognition. The method collects electrical signals of facial and neck muscle activity during a user's mouth movement and facial images during the mouth movement; extracts features from the myoelectric signals and facial images, performs multimodal fusion of the myoelectric signal features with the facial movement features, constructs a joint feature vector, and captures the intrinsic correlation between the muscle activity of silent speech and the mouth movement; constructs a multimodal silent speech recognition model based on a Transformer network to recognize the corresponding speech content; and generates a complete sentence output based on NLP fuzzy language processing. Through the above steps, the speech recognition method based on human mouth movement of the present invention can recognize speech expressed in a silent manner, avoid environmental interference, is convenient for use by people with pronunciation disorders, and is conducive to the confidentiality of communication.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition, and in particular to a speech recognition method and device based on human mouth movement. Background Art

[0002] Acoustic speech signals travel through the air and are therefore susceptible to interference from environmental noise. For example, in crowded places such as restaurants and airports, the performance of speech recognizers can degrade significantly. Traditional speech needs to be clear and audible, especially when it needs to be transmitted or processed through technical systems. In quiet places such as libraries and conferences, this speech can disturb people around and may affect the confidentiality of communications. This is particularly evident, for example, in services that transmit private information. At the same time, for people with speech disorders, such as vocal cord resection patients or individuals with facial paralysis due to stroke, people with speech disorders may be excluded from interpersonal voice communication and voice-driven human-computer interaction. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, one of the purposes of the present invention is to provide a speech recognition method based on human mouth movement that can recognize speech expressed in silence, avoid environmental interference, and is beneficial for people with speech disorders.

[0004] In order to overcome the shortcomings of the prior art, the second purpose of the present invention is to provide a speech recognition device based on human mouth movements that can recognize speech expressed in silence, avoid environmental interference, and is beneficial for people with speech disorders to use.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] A method for language recognition based on human mouth movement, comprising the following steps:

[0007] Data collection: Collect electrical signals of facial and neck muscle activity during the user's mouth movement, as well as facial images during mouth movement;

[0008] Image feature extraction: A facial keypoint detection algorithm is used to accurately locate the mouth and jaw areas in facial images and extract regions of interest. A convolutional neural network is used to extract the geometric features of the mouth in a single frame image. A time series model is used to capture the motion characteristics of the mouth in the temporal dimension. The optical flow method is combined to analyze the motion trajectory between consecutive frames to obtain relevant features. The dynamic change characteristics are analyzed using time series analysis.

[0009] Feature fusion: Multimodal fusion of EMG features and facial movement features to construct a joint feature vector that captures the intrinsic correlation between muscle activity and mouth movements in silent speech.

[0010] Building a multimodal silent speech recognition model based on a Transformer network: This model conducts in-depth analysis and classification of joint feature vectors from multiple input sources to identify the corresponding speech content.

[0011] NLP-based fuzzy language processing: Perform text error detection and candidate generation on the recognized speech content. For the detected erroneous candidate words, design a correction algorithm based on fuzzy matching to generate multiple possible candidate correction schemes. During the candidate correction process, use dependency syntactic analysis and semantic embedding comparison methods to generate complete sentence output.

[0012] Furthermore, in the data collection step, electrical signals of muscle activity are collected by arranging surface electrode arrays at key muscle groups on the face and neck.

[0013] Furthermore, the language recognition method based on human mouth movement also includes an electrical signal processing step, which is located after the data acquisition step and before the feature fusion step. The electrical signal processing step is specifically: removing the power frequency noise and high-frequency interference in the electromyographic signal through a bandpass filter, and at the same time normalizing the signal to extract the features of a specific frequency band to highlight the muscle activity related to pronunciation.

[0014] Furthermore, in the image feature extraction step, before using the facial key point detection algorithm to accurately locate the mouth and jaw areas of the facial image, a deep learning model is used to perform high-precision face detection to ensure accurate positioning of the facial area.

[0015] Furthermore, in the image feature extraction step, a facial key point detection algorithm is used to accurately locate the mouth and jaw areas of the facial image and extract the region of interest. Image segmentation technology is used to remove background interference and only retain mouth-related information. The video data is standardized, including unifying the resolution, frame rate and brightness, to enhance the robustness and consistency of subsequent processing.

[0016] Furthermore, in the image feature extraction step, the optical flow method is combined to analyze the motion trajectory between consecutive frames to obtain relevant features, specifically: geometric parameters of mouth movement are extracted through trajectory analysis, and the geometric parameters include at least one of the lip opening and closing amplitude, closing speed, and upper and lower lip distance.

[0017] Furthermore, in the NLP-based fuzzy language processing step, text error detection and candidate generation for the recognized speech content are specifically performed as follows: error detection is performed using at least one of a statistical language model, a deep learning model, and a vocabulary filtering method. The statistical language model is specifically: a probability distribution model based on N-gram grammar, combined with word frequency and contextual conditional probability, to identify possible spelling or recognition errors; the deep learning model is specifically: using a BERT pre-trained language model to detect context-inconsistent words or phrases through a masked language modeling task; the vocabulary filtering is specifically: combining a domain dictionary or a custom vocabulary for a specific task, and directly marking words that are not within the vocabulary range as suspected errors.

[0018] Furthermore, in the NLP-based fuzzy language processing step, a correction algorithm based on fuzzy matching is designed for the detected erroneous candidate words, and multiple possible candidate correction schemes are generated, specifically: correction is performed using at least one of an edit distance algorithm, context matching optimization, and phoneme similarity matching. The edit distance algorithm is specifically: by calculating the minimum edit distance between the erroneous word and the legal word in the dictionary, the closest correction candidate is generated; the context matching optimization is specifically: using context features to generate candidate words in the wrong position through a model to ensure that the generated results are semantically reasonable in the context; the phoneme similarity matching is specifically: for the problem of misrecognition of homophones or near-phonetic words caused by speech, a phoneme-based similarity matching model is constructed, combined with international phonetic symbols or phoneme vector embedding, to generate candidate words with pronunciations similar to the erroneous words.

[0019] Furthermore, in the NLP-based fuzzy language processing step, dependency syntactic analysis is: using dependency parsing technology to analyze the syntactic structure to ensure that the corrected words are consistent with the subject-verb-object relationship, modifiers, etc. of the entire sentence; semantic embedding comparison is: using the sentence embedding model to map the original sentence and the candidate sentence into the semantic space, and evaluating the semantic consistency of the corrected sentence through cosine similarity.

[0020] The second object of the present invention is achieved by adopting the following technical solution:

[0021] A speech recognition device based on human mouth movement, used to implement any one of the above-mentioned speech recognition methods based on human mouth movement, the speech recognition device based on human mouth movement comprising

[0022] Electrode arrays are placed at key muscle groups on the face and neck to collect electrical signals from muscle activity;

[0023] An image acquisition device, wherein the image acquisition device acquires facial images during mouth movement;

[0024] A processor processes and analyzes the electromyographic signal and the facial image to recognize and generate a complete sentence output.

[0025] Compared with the existing technology, the speech recognition method based on human mouth movement of the present invention collects the electrical signals of the muscle activities of the face and neck during the user's mouth movement and the facial image during the mouth movement; extracts features of the electromyographic signals and facial images, multimodally fuses the electromyographic signal features with the facial movement features, constructs a joint feature vector, and captures the intrinsic correlation between the muscle activity and mouth movements of silent speech; constructs a multimodal silent speech recognition model based on the Transformer network to identify the corresponding speech content; and generates a complete sentence output based on NLP fuzzy language processing. Through the above steps, the speech recognition method based on human mouth movement of the present invention can recognize speech expressed in silence, avoid environmental interference, is convenient for people with pronunciation disorders to use, and is conducive to the confidentiality of communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of the language recognition method based on human mouth movement of the present invention;

[0027] Figure 2 is a structural diagram of the electrode array;

[0028] Figure 3 Schematic diagram of the attachment position of the electrode array;

[0029] Figure 4 Schematic diagram of facial key points.

[0030] In the figure: 10, silicone base; 20, hydrogel; 30, silver chloride circuit. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0033] Due to human respiration, the position of the target on the human body surface will move in real time, resulting in inaccurate medical operations. The present application provides a language recognition method based on human mouth movement for predicting the real-time position of the target on the human body surface, which is used for determining the point during human surgery, such as puncture point, skin cancer radiotherapy point, etc.

[0034] See also Figure 1 The present invention provides a speech recognition method based on human mouth movement, comprising the following steps:

[0035] Data collection: Collect electrical signals of facial and neck muscle activity during the user's mouth movement, as well as facial images during mouth movement;

[0036] Image feature extraction: A facial keypoint detection algorithm is used to accurately locate the mouth and jaw areas in facial images and extract regions of interest. A convolutional neural network is used to extract the geometric features of the mouth in a single frame image. A time series model is used to capture the motion characteristics of the mouth in the temporal dimension. The optical flow method is combined to analyze the motion trajectory between consecutive frames to obtain relevant features. The dynamic change characteristics are analyzed using time series analysis.

[0037] Feature fusion: Multimodal fusion of EMG features and facial movement features to construct a joint feature vector that captures the intrinsic correlation between muscle activity and mouth movements in silent speech.

[0038] Building a multimodal silent speech recognition model based on a Transformer network: This model conducts in-depth analysis and classification of joint feature vectors from multiple input sources to identify the corresponding speech content.

[0039] NLP-based fuzzy language processing: Perform text error detection and candidate generation on the recognized speech content. For the detected erroneous candidate words, design a correction algorithm based on fuzzy matching to generate multiple possible candidate correction schemes. During the candidate correction process, use dependency syntactic analysis and semantic embedding comparison methods to generate complete sentence output.

[0040] The specific steps of data collection are:

[0041] High-precision surface electrode arrays are placed on key muscle groups on the face and neck (such as the zygomatic major, depressor labii, and mentalis muscles) to collect electrical signals from muscle activity. Conductive gel is used to reduce the impedance between the electrode and the skin to ensure signal stability and sensitivity. The specific structure of the electrode array is as follows: Figure 2As shown, it includes a silicone base 10, multiple hydrogels 20, and silver chloride lines 30 extending from the hydrogels 20. Due to the differences in height and body shape of users, the muscle group positions of different users cannot be exactly the same. The electrode array in this application is a flexible sensor. It uses silicone as a base with deformability. It can be deformed within a certain range and has better adhesion and biocompatibility. For the muscle groups active during lip movement, the muscle group collection positions selected by this patent are as follows: Figure 3 shown.

[0042] For facial key point recognition, a high-resolution camera or 3D depth sensor is used to capture facial images, and a facial key point detection algorithm (such as Opencv's dlib or MediaPipe framework) is used to accurately calibrate the spatial position and dynamic changes of the mouth, jaw, and main facial feature points. Figure 4 shown.

[0043] The language recognition method based on human mouth movement also includes an electrical signal processing step, which is located after the data acquisition step and before the feature fusion step. The electrical signal processing step is specifically: removing the power frequency noise and high-frequency interference in the electromyographic signal through a bandpass filter, and at the same time normalizing the signal to extract the features of a specific frequency band to highlight the muscle activity related to pronunciation.

[0044] The specific steps of image feature extraction are:

[0045] First, high-precision face detection is performed based on deep learning models (such as MTCNN or RetinaFace) to ensure accurate positioning of the facial area. Subsequently, a facial key point detection algorithm (such as Dlib or MediaPipe) is used to accurately locate the mouth and jaw areas and extract the region of interest (ROI). In order to further improve the accuracy of the analysis and avoid interference from irrelevant factors, image segmentation technology is applied to remove background interference and only retain mouth-related information. At the same time, the video data is standardized, including unified resolution, frame rate and brightness, to enhance the robustness and consistency of subsequent processing. In the analysis stage of key point coordinates, trajectory analysis is used to extract the geometric parameters of mouth movement, including lip opening and closing amplitude, closing speed, upper and lower lip distance, etc., and time series analysis is used to deeply explore dynamic change characteristics.

[0046] In addition to extracting facial key points, we use a convolutional neural network to extract geometric features of the mouth (such as mouth outline and degree of opening and closing) from a single frame. We also use a temporal model (Transformer) to capture the temporal motion of the mouth. We also combine this with optical flow to analyze the motion trajectory between consecutive frames and obtain relevant features.

[0047] The specific steps for building a multimodal silent speech recognition model based on the Transformer network are as follows:

[0048] A multimodal silent speech recognition model is constructed based on the Transformer network, which fully utilizes its advantages in time series modeling, conducts in-depth analysis and classification of joint features from multiple source inputs (such as surface electromyography signals and facial key point trajectories, mouth contours), and accurately identifies the corresponding speech content. For different specific tasks (such as pronunciation detection, lip shape category recognition, etc.), special classification modules are designed to optimize the model structure to adapt to diverse application needs. During the model development process, the rationality of model parameter initialization is improved through multi-stage pre-training, combining existing public data sets with data collected by custom devices, thereby improving the ability to capture complex dynamic relationships in multimodal signals. During the training phase, data enhancement and optimization strategies are further adopted to ensure that the model has higher robustness and generalization capabilities when dealing with non-standardized inputs, noise interference, and cross-domain migration.

[0049] The specific steps of fuzzy language processing based on NLP are:

[0050] To improve the accuracy of the post-processing phase of text generated by the speech recognition model and address the issue of ambiguous text caused by fuzzy speech recognition, we designed an algorithm for ambiguous text recognition and correction. This algorithm was refined across multiple levels, including text error detection, fuzzy matching, semantic understanding, multimodal support, and result optimization.

[0051] Text error detection and candidate generation for recognized speech content specifically involves using at least one of statistical language models, deep learning models, and vocabulary filtering to detect errors. The statistical language model specifically uses a probability distribution model based on N-gram grammar, combining word frequency and contextual probabilities to identify possible spelling or recognition errors. For example, low-probability words are marked as error candidates. The deep learning model specifically uses the BERT pre-trained language model to detect context-inconsistent words or phrases through the masked language modeling task. Vocabulary filtering specifically combines a domain dictionary or a custom vocabulary for a specific task to directly mark words outside the vocabulary as suspected errors.

[0052] For detected incorrect candidate words, a correction algorithm based on fuzzy matching is designed to generate multiple possible candidate correction solutions. Specifically, correction is performed using at least one of the following methods: edit distance algorithm, contextual matching optimization, and phoneme similarity matching. The edit distance algorithm calculates the minimum edit distance between the incorrect word and a valid word in the dictionary to generate the closest correction candidate. Combined with weighted edit distance, the weights of insertion, deletion, and replacement operations are dynamically adjusted to adapt to specific language or domain characteristics. Contextual matching optimization utilizes contextual features to generate candidate words for the incorrect position through a model, ensuring that the generated results are semantically reasonable in the context. For example, the optimal candidates are selected through word score ranking generated by BERT. Phoneme similarity matching addresses the problem of misidentification of homophones or near-phonetic words caused by speech. A phoneme-based similarity matching model is constructed, combined with the International Phonetic Alphabet or phoneme vector embeddings, to generate candidate words with similar pronunciation to the incorrect word.

[0053] In the fuzzy language processing step based on NLP, it is crucial to consider the semantic structure and grammatical rationality of the entire sentence during the candidate correction process. To this end, a multi-layer semantic understanding module is designed:

[0054] Dependency syntactic analysis is to use dependency parsing technology to analyze the syntactic structure to ensure that the corrected words are consistent with the subject, predicate, object relationship, modifiers, etc. of the entire sentence.

[0055] Semantic embedding comparison is as follows: the original sentence and candidate sentences are mapped into the semantic space using the sentence embedding model, and the semantic consistency of the corrected sentences is evaluated by cosine similarity.

[0056] Finally, a generative language model (GPT series) is used to generate complete sentences, ensuring the grammatical and semantic fluency of the output text.

[0057] The present invention also discloses a speech recognition device based on human mouth movement, which is used to implement any of the above-mentioned speech recognition methods based on human mouth movement. The speech recognition device based on human mouth movement includes:

[0058] Electrode arrays are placed on key muscle groups on the face and neck to collect electrical signals from muscle activity;

[0059] An image acquisition device, wherein the image acquisition device acquires facial images during the movement of the mouth;

[0060] The processor processes and analyzes the electromyographic signals and facial images to recognize and generate complete sentence output.

[0061] Compared with the existing technology, the speech recognition method based on human mouth movement of the present invention collects the electrical signals of the muscle activities of the face and neck during the user's mouth movement and the facial image during the mouth movement; extracts features of the electromyographic signals and facial images, multimodally fuses the electromyographic signal features with the facial movement features, constructs a joint feature vector, and captures the intrinsic correlation between the muscle activity and mouth movements of silent speech; constructs a multimodal silent speech recognition model based on the Transformer network to identify the corresponding speech content; and generates a complete sentence output based on NLP fuzzy language processing. Through the above steps, the speech recognition method based on human mouth movement of the present invention can recognize speech expressed in silence, avoid environmental interference, is convenient for people with pronunciation disorders to use, and is conducive to the confidentiality of communication.

[0062] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patented invention. It should be noted that those skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention. These variations and improvements are equivalent modifications and improvements to the above embodiments based on the essential technology of the present invention and fall within the scope of protection of the present invention.

Claims

1. A speech recognition method based on human mouth movement, characterized in that: The following steps are involved: Data collection: Collect electrical signals of facial and neck muscle activity during the user's mouth movement, as well as facial images during mouth movement; Image feature extraction: A facial keypoint detection algorithm is used to accurately locate the mouth and jaw areas in facial images and extract regions of interest. A convolutional neural network is used to extract the geometric features of the mouth in a single frame image. A time series model is used to capture the motion characteristics of the mouth in the temporal dimension. The optical flow method is combined to analyze the motion trajectory between consecutive frames to obtain relevant features. The dynamic change characteristics are analyzed using time series analysis. Feature fusion: Multimodal fusion of EMG features and facial movement features to construct a joint feature vector that captures the intrinsic correlation between muscle activity and mouth movements in silent speech. Building a multimodal silent speech recognition model based on a Transformer network: This model conducts in-depth analysis and classification of joint feature vectors from multiple input sources to identify the corresponding speech content. NLP-based fuzzy language processing: Perform text error detection and candidate generation on the recognized speech content. For the detected erroneous candidate words, design a correction algorithm based on fuzzy matching to generate multiple possible candidate correction schemes. During the candidate correction process, use dependency syntactic analysis and semantic embedding comparison methods to generate complete sentence output.

2. The method for speech recognition based on human mouth movement according to claim 1, wherein: In the data collection step, electrical signals of muscle activity are collected by arranging surface electrode arrays at key muscle groups on the face and neck.

3. The method for speech recognition based on human mouth movement according to claim 1, wherein: The language recognition method based on human mouth movement also includes an electrical signal processing step, which is located after the data acquisition step and before the feature fusion step. The electrical signal processing step specifically includes: removing power frequency noise and high-frequency interference in the electromyographic signal through a bandpass filter, normalizing the signal at the same time, and extracting features of a specific frequency band to highlight muscle activity related to pronunciation.

4. The method for speech recognition based on human mouth movement according to claim 1, wherein: In the image feature extraction step, before using the facial key point detection algorithm to accurately locate the mouth and jaw areas of the facial image, a deep learning model is used to perform high-precision face detection to ensure accurate positioning of the facial area.

5. The method for speech recognition based on human mouth movement according to claim 1, wherein: In the image feature extraction step, a facial key point detection algorithm is used to accurately locate the mouth and jaw areas of the facial image and extract the region of interest. Image segmentation technology is used to remove background interference and only retain information related to the mouth. The video data is standardized, including unifying the resolution, frame rate and brightness, to enhance the robustness and consistency of subsequent processing.

6. The method for speech recognition based on human mouth movement according to claim 1, wherein: In the image feature extraction step, the motion trajectory between consecutive frames is analyzed in combination with the optical flow method to obtain relevant features. Specifically, the geometric parameters of the mouth movement are extracted through trajectory analysis. The geometric parameters include at least one of the lip opening and closing amplitude, closing speed, and upper and lower lip distance.

7. The method for speech recognition based on human mouth movement according to claim 1, wherein: In the NLP-based fuzzy language processing step, text error detection and candidate generation for the recognized speech content are specifically performed as follows: error detection is performed using at least one of a statistical language model, a deep learning model, and a vocabulary filtering method. The statistical language model is specifically: a probability distribution model based on N-gram grammar, combined with word frequency and contextual conditional probability, to identify possible spelling or recognition errors; the deep learning model is specifically: using the BERT pre-trained language model to detect context-inconsistent words or phrases through a masked language modeling task; the vocabulary filtering is specifically: combining a domain dictionary or a custom vocabulary for a specific task, and directly marking words that are not within the vocabulary range as suspected errors.

8. The method for speech recognition based on human mouth movement according to claim 1, wherein: In the NLP-based fuzzy language processing step, a correction algorithm based on fuzzy matching is designed for the detected erroneous candidate words, and multiple possible candidate correction schemes are generated. Specifically, correction is performed using at least one of an edit distance algorithm, context matching optimization, and phoneme similarity matching. The edit distance algorithm is specifically: by calculating the minimum edit distance between the erroneous word and the legal word in the dictionary, the closest correction candidate is generated; the context matching optimization is specifically: using context features to generate candidate words in the wrong position through a model to ensure that the generated results are semantically reasonable in the context; the phoneme similarity matching is specifically: for the problem of misrecognition of homophones or near-phonetic words caused by speech, a phoneme-based similarity matching model is constructed, combined with international phonetic symbols or phoneme vector embedding, to generate candidate words with pronunciations similar to the erroneous words.

9. The method for speech recognition based on human mouth movement according to claim 1, wherein: In the NLP-based fuzzy language processing step, dependency syntactic analysis is to use dependency parsing technology to analyze the syntactic structure to ensure that the corrected words are consistent with the subject-verb-object relationship and modifiers of the entire sentence; Semantic embedding comparison is as follows: the original sentence and candidate sentences are mapped into the semantic space using the sentence embedding model, and the semantic consistency of the corrected sentences is evaluated by cosine similarity.

10. A speech recognition device based on human mouth movement, used to implement the speech recognition method based on human mouth movement according to any one of claims 1 to 9, characterized in that: The speech recognition device based on human mouth movement includes Electrode arrays are placed at key muscle groups on the face and neck to collect electrical signals from muscle activity; An image acquisition device, wherein the image acquisition device acquires facial images during mouth movement; A processor processes and analyzes the electromyographic signal and the facial image to recognize and generate a complete sentence output.

Images