A speech decoding and recognition method and system for multimodal laryngeal vibration signals and lip motion point data

Abstract

The present invention discloses a speech decoding and recognition method and system for multi-modal laryngeal vibration signals and lip motion point data. The present invention decodes intonation and the four tones of Chinese by collecting laryngeal vibration signals during utterance, and decodes consonants, vowels and syllables by collecting lip motion point data during utterance. The results of the two are multi-modally fused, and the results are sent to a large language model, and finally the large language model outputs recognized text. Compared with the traditional face and neck physiological signal speech decoding method, the speech recognition method of the present invention has the characteristics of being suitable for long-term monitoring, conforming to the syllable composition principle of linguistics, being less affected by the corpus, having strong generalization ability, and having single-character level recognition precision and accuracy that cannot be achieved by a single modality.

Description

Technical Field

[0001] The present invention belongs to the field of biological signal processing, and in particular relates to a speech decoding and recognition method and system for multi-modal laryngeal vibration signals and lip motion point data. Background Art

[0002] Speech is the most important way for humans to communicate and is also a basic survival skill. Speech recognition is a speech technology in daily life that can help computers understand the speaker's intentions. Its development has greatly promoted the relationship between humans and computers and facilitated natural and efficient human-computer interaction.

[0003] The existing commonly used speech decoding method is to use a high-density electrode array to collect surface electromyography signals and decode the surface electromyography signals to achieve speech recognition.

[0004] However, there are still problems and limitations in actual application scenarios. The amplitude of the surface electromyographic signal is weak, which makes it difficult to obtain, and is easily affected by factors such as muscle state and sensor position, resulting in reduced accuracy of speech recognition; the high-density electrode array is not wearable and comfortable enough, and the operation of collecting signals is relatively complicated and cumbersome, and it is not suitable for long-term signal collection. The increase in data dimensions caused by the increase in channels has greatly increased the complexity of subsequent calculations and processing; existing decoding methods mostly rely on label-aligned training data, and model training is complex, which increases the technical barriers to speech decoding. At the same time, existing decoding methods are unable to decode tones and the four tones of Chinese, which limits the accuracy of decoding, and currently cannot achieve accurate decoding at the single-character level.

[0005] Therefore, it is necessary to provide a device for decoding speech to solve the above technical problems.

[0006] Invention patent content

[0007] The present invention provides a speech decoding and recognition method of multi-modal laryngeal vibration signals and lip motion point data to solve the technical problems existing in the known technology.

[0008] A speech decoding and recognition method for multi-modal laryngeal vibration signals and lip motion point data comprises the following steps:

[0009] Step 1, collecting laryngeal vibration signals and lip moving point data;

[0010] Step 2, preprocessing the collected laryngeal vibration signal to obtain a preprocessed laryngeal vibration signal;

[0011] Step 3, inputting the preprocessed laryngeal vibration signal into a laryngeal vibration signal recognition model, extracting the time-frequency domain features of the laryngeal vibration signal, processing the extracted time-frequency domain features and outputting the tone recognition result of the laryngeal vibration signal;

[0012] The lip motion data is input into the lip motion data recognition model. The temporal and spatial features of the lip motion data are first encoded using the Conformer encoding network. Then, the encoded content is decoded simultaneously by the Transformer decoder and the CTC decoder. Finally, the weights are combined and the SoftMax layer is used to output the syllable recognition results of the lip motion data.

[0013] Step 4: obtaining a set of tone recognition results of a single category of laryngeal vibration signals from the laryngeal vibration signal recognition model, and further obtaining probability distribution values ​​of various categories of tones of the laryngeal vibration signals; obtaining a set of syllable recognition results of a single category of lip motion point data from the lip motion point data recognition model, and further obtaining probability distribution values ​​of various categories of syllables of the lip motion point data;

[0014] Step 5, the probability distribution values ​​of each category of the tone of the laryngeal vibration signal and the probability distribution values ​​of each category of the syllable of the lip movement point data are synchronously input into the multimodal fusion model, the multimodal fusion model merges the two inputs of tone and syllable, the multimodal fusion model outputs the synthetic speech probability matrix, synthesizes the complete syllable, and sends it to the large language model, the output result of the large language model is the final speech recognition result.

[0015] In the present invention, compared with the traditional speech decoding method based on facial and neck physiological signals, the lip moving point data decoding method in step 3 and step 4 does not require contact, and the contact of laryngeal vibration signal decoding also has a smaller contact area than electromyographic signal decoding and does not require the addition of conductive gel. Both have better comfort and are suitable for long-term monitoring of speech. Existing decoding methods mostly rely on label-aligned training data corpora, and model training is complex, which increases the technical barriers to speech decoding. At the same time, the existing decoding methods cannot decode tones and the four tones of Chinese, which limits the accuracy of decoding, and currently cannot achieve accurate decoding at the single character level. A speech decoding and recognition method for multimodal laryngeal vibration signals and lip moving point data, step 5, conforms to the composition rules of phonemes and syllables in linguistics, so that the speech recognition method is less affected by the corpus and can be directly migrated after model optimization. And it can achieve single-character level recognition accuracy and higher recognition accuracy that traditional facial and neck speech decoding methods cannot achieve.

[0016] In step 1, the laryngeal vibration signal includes: a vocal cord vibration signal related to phonation, and a vibration signal generated when the neck muscles contract;

[0017] The lip moving point data includes: movement data of the lips and the face around the lips when speaking.

[0018] In step 2, the preprocessing includes signal amplification, filtering and Fourier transform.

[0019] In step 3, the laryngeal vibration signal recognition model includes: a time-frequency domain feature extraction module and a support vector machine classification module connected in sequence;

[0020] The time-frequency domain feature extraction module is used to extract the time-frequency domain features of the laryngeal vibration signal;

[0021] The support vector machine classification module is used to process the laryngeal vibration signal and output the tone recognition result of the laryngeal vibration signal.

[0022] In step 3, the time-frequency domain characteristics of the laryngeal vibration signal include the number of zero crossings.

[0023] In step 5, the multimodal fusion model is a common model such as TFN (Multimodal Tensor Fusion Network), LMF (Low-rank Multimodal Fusion), PTP (polynomial tensor pooling), etc.

[0024] A speech decoding and recognition system for multi-modal laryngeal vibration signals and lip motion point data, comprising:

[0025] A laryngeal vibration signal acquisition module, a laryngeal vibration signal preprocessing module, and a laryngeal vibration signal recognition model connected in sequence;

[0026] A depth camera, a lip motion point data preprocessing module, and a lip motion point data recognition model connected in sequence;

[0027] A multimodal fusion model, wherein the input end of the multimodal fusion model is connected to the output end of the laryngeal vibration signal recognition model and the output end of the lip moving point data recognition model;

[0028] A large language model connected to the output end of the multimodal fusion model;

[0029] The laryngeal vibration signal acquisition module is used to acquire laryngeal vibration signals generated by vocal cord vibration and neck muscle contraction related to phonation, the laryngeal vibration signal preprocessing module is used to preprocess the acquired laryngeal vibration signals, and the laryngeal vibration signal recognition model is used to perform feature extraction, tone and Chinese four tones classification and speech prediction on the laryngeal vibration signals of the preprocessed larynx;

[0030] The depth camera is used to collect the movement data of the lips and the face around the lips when speaking, the lip moving point data preprocessing module is used to preprocess the collected lip moving point data, and the lip moving point data recognition model is used to extract features from the preprocessed moving point data and recognize syllables;

[0031] The multimodal fusion model synthesizes the tone results recognized by the laryngeal vibration recognition model and the syllable results recognized by the lip motion data model, outputs the synthesized results, sends them to the large language model, and outputs the final recognition results.

[0032] The laryngeal vibration signal acquisition module comprises one or more signal acquisition channels, and each acquisition channel comprises a piezoelectric sensor for acquiring laryngeal vibration signals.

[0033] The laryngeal vibration signal preprocessing module comprises a signal amplifier, a filter and a Fourier transformer.

[0034] The technical solution adopted by the present invention to solve the technical problems existing in the known technology is: a speech recognition system of multimodal laryngeal vibration signal and lip moving point data, comprising: a laryngeal vibration signal acquisition system, a laryngeal vibration signal preprocessing system, and a laryngeal vibration signal recognition model connected in sequence; a depth camera, a lip moving point data preprocessing system, and a lip moving point data recognition model connected in sequence; and a multimodal fusion model, wherein the input end of the multimodal fusion model is respectively connected to the output end of the laryngeal vibration signal recognition model and the lip moving point data recognition model;

[0035] The laryngeal vibration signal acquisition system is used to acquire laryngeal vibration signals generated by vocal cord vibration and neck muscle contraction related to phonation. The laryngeal vibration signal preprocessing system is used to preprocess the acquired laryngeal vibration signals. The laryngeal vibration signal recognition model is used to extract features of the laryngeal vibration signals of the preprocessed larynx, classify the tones and four Chinese tones, and perform speech prediction.

[0036] The depth camera is used to collect the movement data of the lips and the surrounding face when speaking, the lip moving point data preprocessing system is used to preprocess the collected lip moving point data, and the lip moving point data recognition model is used to extract features from the preprocessed moving point data and recognize syllables;

[0037] The multimodal model synthesizes the tone results recognized by the laryngeal vibration recognition model and the syllable results recognized by the lip movement data model, outputs the synthesized results, and sends them to the large language model to output the final recognition results.

[0038] Furthermore, the laryngeal vibration signal recognition model includes a time-frequency domain feature extraction module and a support vector machine classification module which are connected in sequence; the time-frequency domain feature extraction module is used to extract the time-frequency domain features of the laryngeal vibration signal, and the support vector machine classification module is used to process the feature information and modeling of the laryngeal vibration signal, and output the speech classification result of the laryngeal vibration signal.

[0039] Furthermore, the lip reading recognition model includes a Conformer encoding network, a Transformer decoder and a CTC (Connectionist temporal classification) decoder connected in sequence. Based on the moving point data collected by Live Link Face, the lip reading recognition model uses a fully connected Conformer network to encode the temporal and spatial features of the moving points, and its Transformer decoder and CTC decoder decode the encoded content at the same time, and after combining the weights, it is used to output the speech prediction of the lip moving point data through the SoftMax layer.

[0040] Furthermore, the laryngeal vibration signal acquisition system includes one or more signal acquisition channels, each acquisition channel includes a piezoelectric sensor for collecting laryngeal vibration signals, and the multiple piezoelectric sensors are respectively arranged at different positions.

[0041] Furthermore, the laryngeal vibration signal preprocessing system includes a signal amplifier, a filter and a Fourier transformer; the signal amplifier is used to amplify the laryngeal vibration signal collected by the laryngeal vibration sensor, the filter is used to filter the amplified laryngeal vibration signal, and the Fourier transformer is used to perform time-frequency domain transformation on the filtered laryngeal vibration signal to facilitate subsequent feature extraction.

[0042] Furthermore, the filter refers to a notch filter, which is used to remove 50 Hz power frequency interference and its harmonic components.

[0043] Furthermore, the lip motion point data system includes a depth camera; a lip motion point data preprocessing system, which is implemented through Live Link Face, and Live Link Face converts the collected video data into motion point data.

[0044] The present invention comprises: a throat vibration signal acquisition system, a throat vibration signal preprocessing system, and a throat vibration signal recognition model connected in sequence; a lip moving point data acquisition system, a lip moving point data preprocessing system, and a lip moving point data recognition model connected in sequence; a multimodal fusion model and a large language model; the input end of the multimodal fusion model is respectively connected to the lip moving point data recognition model and the output end of the throat vibration signal recognition, and is used for synthesizing the tone classified by the throat vibration signal recognition model and the four tones of Chinese and the syllable recognition result of the lip moving point data recognition model, and outputting the complete pinyin syllable to the large language model, and finally the large language model outputs the recognized text.

[0045] The method for speech recognition of multimodal laryngeal vibration signals and lip moving point data of the speech recognition system using the multimodal laryngeal vibration signals and lip moving point data comprises the following steps:

[0046] Step 1: Collect and create training sample and test sample data sets; and train and test the laryngeal vibration signal recognition model and the lip moving point data recognition model respectively;

[0047] Step 2: The laryngeal vibration signal acquisition system and the lip moving point data acquisition system synchronously acquire the laryngeal vibration signal and the lip moving point data;

[0048] Step 3, the laryngeal vibration signal preprocessing system and the lip moving point data recognition preprocessing system perform preprocessing on the collected laryngeal vibration signal and lip moving point data signal;

[0049] Step 4, independently predict the laryngeal vibration signal and lip moving point data of the pre-processed larynx, and input the trained laryngeal vibration signal recognition model and lip moving point data recognition model accordingly; wherein, the laryngeal vibration signal recognition model obtains a score vector of a single category of a set of laryngeal vibration signals, and further obtains a set of probability distribution values ​​of each category of the laryngeal vibration signals; the lip moving point data recognition model obtains a score vector of a single category of a set of continuous lip moving point data, and further obtains a set of probability distribution values ​​of each category of the set of continuous lip moving point data;

[0050] Step 5, the category probability distribution value of the laryngeal vibration signal and the category probability distribution value of the lip moving point data are synchronously input into the multimodal fusion model in the form of vectors. The multimodal fusion model synthesizes these two columns of vectors to obtain a probability matrix as the complete syllable recognition result.

[0051] The advantages and positive effects of the present invention are:

[0052] The silent speech recognition method of the present invention integrates the laryngeal vibration signal and the lip moving point data, synchronously collects the laryngeal vibration signal and the lip moving point data and performs multimodal fusion. Compared with the traditional recognition method using single-mode information such as electromyographic signals or lip images as input, the present invention has significantly improved accuracy and can achieve single-character level speech recognition accuracy that single-mode information cannot achieve.

[0053] The speech recognition method of multimodal laryngeal vibration signals and lip motion point data proposed in the present invention has excellent recognition accuracy, is less restricted by the corpus text library, is suitable for long-term use by patients with speech disorders, is easy to operate, and has single-character level recognition accuracy that cannot be achieved by recognition methods that only use single-modal information as input. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a structural schematic diagram of a speech recognition system of a multi-modal laryngeal vibration signal and lip motion point data of the present invention.

[0055] Figure 2It is a schematic diagram of the working process of a laryngeal vibration signal recognition model of the present invention.

[0056] Figure 3 It is a schematic diagram of the workflow of a lip moving point data recognition model of the present invention.

[0057] Figure 4 This is a working flow chart of a speech recognition method of a multi-modal laryngeal vibration signal and lip moving point data of the present invention. DETAILED DESCRIPTION

[0058] In order to further understand the content, features and effects of the present invention, the following embodiments are listed and described in detail with reference to the accompanying drawings:

[0059] The Chinese meanings of the English words and abbreviations in this article are as follows:

[0060] SoftMax: Softmax function, also known as normalized exponential function. It is a generalization of the binary classification function sigmoid to multi-classification, and its purpose is to present the results of multi-classification in the form of probability.

[0061] Live Link Face: Live Link Face is an application that can extract the facial features of the subject through a depth camera and convert them into moving point data for tracking.

[0062] Figure 1 The framework of a speech recognition system for multimodal laryngeal vibration signals and lip motion point data is shown, including: the laryngeal vibration signal is collected and preprocessed, and then sent to the laryngeal vibration signal recognition model to realize the classification and recognition of tones and the four tones of Chinese; the lip motion point data is collected and preprocessed, and then sent to the lip motion point data recognition model to realize the classification and recognition of syllables; and a fusion model and a large language model; the input end of the fusion model is connected to the output ends of the lip motion point data recognition model and the laryngeal vibration signal recognition model respectively, and the output end of the fusion model is connected to the input end of the large language model;

[0063] Figure 2 The structure of the laryngeal vibration signal recognition model is shown. After the signal is input, it first passes through the time-frequency domain feature extraction module. The extracted time-frequency domain features are sent to the machine learning classification and the classification recognition results are output.

[0064] Figure 3 The structure of the lip motion data recognition model is shown. After the signal is input, it first passes through the Conformer encoding network to encode the temporal and spatial features. The Transformer decoder and CTC decoder decode the encoded content at the same time. Finally, the weights are combined and used to output the syllable recognition results of the lip motion data through the SoftMax layer.

[0065] Figure 4 What is shown is the implementation process of a preferred embodiment of the present invention.

[0066] A speech recognition system for multimodal laryngeal vibration signals and lip moving point data, comprising: a laryngeal vibration signal acquisition system, a laryngeal vibration signal preprocessing system, and a laryngeal vibration signal recognition model connected in sequence; a lip moving point data acquisition system, a lip moving point data preprocessing system, and a lip moving point data recognition model connected in sequence; and a multimodal fusion model and a large language model; the input end of the multimodal fusion model is respectively connected to the output ends of the lip moving point data recognition model and the laryngeal vibration signal recognition model, and the output end of the multimodal fusion model is connected to the input end of the large language model;

[0067] The lip moving point data acquisition system is used to collect the movement data of the lips and the face around the lips when speaking, the lip moving point data preprocessing system is used to preprocess the collected lip image data, and the lip moving point data recognition model is used to extract features from the preprocessed lip moving point data and perform speech recognition;

[0068] The multimodal fusion model is used to merge the four-tone prediction results of the laryngeal vibration signal recognition model and the syllable prediction results of the lip movement data recognition model, and output the comprehensive predicted pinyin results, which are sent to the large language model.

[0069] The merging method of the multimodal fusion model can be adopted to multiply the classification probability matrix of the laryngeal vibration signal recognition model and the classification probability matrix of the lip moving point data recognition model to obtain the final classification probability matrix, and use the maximum value as the final classification result to complete the merging of the tone and the syllable, and output the corresponding complete syllable to the large language model. In addition, the multimodal fusion model can also use other multimodal fusion methods in the prior art.

[0070] Preferably, the laryngeal vibration signal recognition model may include a time-frequency domain feature extraction module and a machine learning classification module in sequence; its time-frequency domain feature extraction module can be used to extract the time-frequency domain features of the laryngeal vibration signal, and its machine learning classification module can be used to process the feature information of the laryngeal vibration module and perform classification, and output the speech classification result of the laryngeal vibration signal.

[0071] Preferably, the lip motion point data recognition model may include a Conformer encoding network, a Transformer decoder and a CTC decoder connected in sequence. The fully connected Conformer network encodes the temporal and spatial features of the motion point, and its Transformer decoder and CTC decoder decode the encoded content at the same time, and after combining the weights, the SoftMax layer is used to output the speech prediction of the lip motion point data.

[0072] Preferably, the laryngeal vibration signal acquisition system may include multiple signal acquisition channels, each acquisition channel may include one or more piezoelectric sensors for collecting laryngeal vibration signals, and the multiple piezoelectric sensors are respectively arranged at different positions.

[0073] Preferably, the laryngeal vibration signal preprocessing system may include a signal amplifier, a filter and a Fourier transformer; the signal amplifier is used to amplify the laryngeal vibration signal collected by the piezoelectric sensor, the filter is used to filter the amplified laryngeal vibration signal, and the Fourier transformer is used to perform time-frequency domain conversion on the filtered vibration signal.

[0074] Preferably, the filter may include a notch filter, and the notch filter is used to remove 50 Hz power frequency interference and its harmonic components.

[0075] Preferably, the lip motion point data system includes a depth camera; the lip motion point data preprocessing system is implemented through LiveLink Face, and Live Link Face converts the data collected by the depth camera into motion point data.

[0076] The present invention also provides a method for speech recognition of multimodal laryngeal vibration signals and lip moving point data of a speech recognition system using the multimodal laryngeal vibration signals and lip moving point data, comprising the following steps:

[0077] Step 1: Collect and create training sample and test sample data sets; and train and test the laryngeal vibration signal recognition model and the lip moving point data recognition model respectively;

[0078] Step 2: The laryngeal vibration signal acquisition system and the lip moving point data acquisition system synchronously acquire the laryngeal vibration signal and the lip moving point data;

[0079] Step 3, the laryngeal vibration signal preprocessing system and the lip moving point data recognition preprocessing system perform preprocessing on the collected laryngeal vibration signal and lip moving point data signal;

[0080] Step 4, independently predicting the laryngeal vibration signal and lip moving point data of the pre-processed larynx corresponding to the laryngeal vibration signal recognition model and the lip moving point data recognition model that have been trained; wherein the laryngeal vibration signal recognition model obtains a score vector of a single category of the tone of a set of laryngeal vibration signals, and further obtains a probability distribution value of each category of the tone of a set of laryngeal vibration signals; the lip moving point data recognition model obtains a score vector of a single category of the syllable of a set of continuous lip moving point data, and further obtains a probability distribution value of each category of the syllable of a set of continuous lip moving point data;

[0081] Step 5, the tone category probability distribution value of the laryngeal vibration signal and the syllable category probability distribution value of the lip movement point data are synchronously input into the multimodal fusion model in the form of vectors. The multimodal fusion model synthesizes these two vectors to obtain the pinyin probability matrix as the recognition result, and sends it to the large language model to finally output the recognized text.

[0082] The following is a preferred embodiment of the present invention to further illustrate the working process and working principle of the present invention:

[0083] 1. Create a dataset

[0084] We designed a data acquisition system, which consists of hardware (piezoelectric sensor, signal amplifier, voltage collector, depth camera, etc.) and software (signal synchronization acquisition program) to synchronously collect laryngeal vibration signals and lip motion point data. We selected the position where the vocal cords and neck are related to voice production and have less interference as the laryngeal vibration signal collection point to place the piezoelectric sensor, using one channel and placed below the Adam's apple. The piezoelectric sensor is used to collect the vocalization action signal of laryngeal vibration. We used a depth camera fixed opposite the subject, and manually adjusted the distance between the camera and the lips to keep the subject's lips completely within the camera screen when speaking, so as to capture the movement characteristics of the subject's lips when speaking. During the data collection experiment, the subject was required to sit on a chair and read aloud according to the text presented by the signal synchronization acquisition program, and avoid unnecessary body movements (shaking head, coughing, yawning, swallowing, etc.). Each collection experiment lasted about 2 hours, including the preliminary preparation of the experiment, data collection, and sufficient rest time for the subject to prevent fatigue during the experiment.

[0085] 2. Preprocessing

[0086] Preprocessing of laryngeal vibration signal: The 50 Hz power frequency interference is removed by a notch filter, and the laryngeal vibration signal in the time domain is converted into a frequency domain signal by short-time Fourier transform. By extracting the time-frequency domain features, six time-frequency spectrum features are obtained, including Mel-Frequency Cepstral Coefficients (MFCCs), mean absolute value (MAV), zero crossing (Zero crossing, ZC), slope signchange (SSC), mean frequency (MNF) and root mean square (RMS), which are all well known to those skilled in the art and will not be described in detail here.

[0087] Live link face converts the video captured by the depth camera into moving point data of lip movement feature points, and inputs it into the fully connected Conformer encoding network to encode the temporal and spatial features of the moving points.

[0088] 3. Prediction results

[0089] First, the laryngeal vibration signal and lip motion point data are input into the corresponding laryngeal vibration signal recognition model and lip motion point data recognition model respectively to independently recognize the tone and syllable. The output of the recognition model needs to be normalized by the SoftMax activation function.

[0090] The SoftMax function formula is:

[0091]

[0092] In the formula, a k represents the kth input signal of the output layer, the denominator represents the total M output signals of the output layer, and calculates the exponential sum of the input signals in all output layers. k is the output of the kth signal, and e is a natural constant.

[0093] In order to combine the tone results recognized by the laryngeal vibration signal and the syllable results recognized by the lip movement data into a complete syllable, the probability vectors obtained by the laryngeal vibration signal recognition model and the lip movement data recognition model are multiplied to obtain the pinyin probability matrix after the product. The specific formula is:

[0094]

[0095] y i is the actual corresponding probability value of pinyin i. is the tone probability value of pinyin i obtained by the surface electromyography signal recognition model; y ic is the syllable probability value of pinyin i obtained by the lip motion data recognition model. The matrix is ​​input into the large language model, and the text result output by the large language model is the final result.

[0096] Laryngeal vibration signal recognition model structure:

[0097] The laryngeal vibration signal recognition model adopts an end-to-end structure, such as Figure 2 As shown in the figure, the original laryngeal vibration signal is preprocessed to obtain the time-frequency domain features and used as the input of the laryngeal vibration signal recognition model. It is classified through machine learning methods such as Linear Discriminant Analysis (LDA) or Support Vector Machine (SVM), and the classification results of the tone are output.

[0098] Lip moving point data recognition model structure:

[0099] like Figure 3 As shown in the figure, after the lip motion point data is processed by the fully connected Conformer encoding network, the re-encoded motion point time-space features are input into the Transformer decoder and CTC decoder. The two encoders decode the encoded content at the same time and output the syllable recognition results of the lip motion point data through the SoftMax layer after combining the weights.

[0100] Comprehensive network structure:

[0101] The network structure that integrates laryngeal vibration information and lip motion point data is as follows: Figure 4 As shown, the laryngeal vibration and lip movement point data are simultaneously input into the laryngeal vibration signal recognition model and the lip movement point data recognition model. After the independent output results of the two models are integrated, the output tone probability vector is multiplied with the syllable probability vector to obtain the final pinyin syllable probability matrix.

[0102] Large language model structure:

[0103] Receive the pinyin syllable probability matrix input of the multimodal fusion model and output the recognized Chinese characters.

[0104] The embodiments described above are only used to illustrate the technical ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The patent scope of the present invention cannot be limited only by this embodiment, that is, any equivalent changes or modifications made to the spirit disclosed by the present invention still fall within the patent scope of the present invention.

Claims

1. A speech decoding and recognition method for multi-modal laryngeal vibration signals and lip motion point data, characterized in that: The steps include: Step 1, collecting laryngeal vibration signals and lip moving point data; Step 2, preprocessing the collected laryngeal vibration signal to obtain a preprocessed laryngeal vibration signal; Step 3, inputting the preprocessed laryngeal vibration signal into a laryngeal vibration signal recognition model, extracting the time-frequency domain features of the laryngeal vibration signal, processing the extracted time-frequency domain features and outputting the tone recognition result of the laryngeal vibration signal; The lip motion data is input into the lip motion data recognition model. The temporal and spatial features of the lip motion data are first encoded using the Conformer encoding network. Then, the encoded content is decoded simultaneously by the Transformer decoder and the CTC decoder. Finally, the weights are combined and the SoftMax layer is used to output the syllable recognition results of the lip motion data. Step 4: obtaining a set of tone recognition results of a single category of laryngeal vibration signals from the laryngeal vibration signal recognition model, and further obtaining probability distribution values ​​of various categories of tones of the laryngeal vibration signals; obtaining a set of syllable recognition results of a single category of lip motion point data from the lip motion point data recognition model, and further obtaining probability distribution values ​​of various categories of syllables of the lip motion point data; Step 5, the probability distribution values ​​of each category of the tone of the laryngeal vibration signal and the probability distribution values ​​of each category of the syllable of the lip movement point data are synchronously input into the multimodal fusion model, the multimodal fusion model merges the two inputs of tone and syllable, the multimodal fusion model outputs the synthetic speech probability matrix, synthesizes the complete syllable, and sends it to the large language model, the output result of the large language model is the final speech recognition result.

2. The method for speech decoding and recognition of multimodal laryngeal vibration signals and lip motion point data according to claim 1, characterized in that: In step 1, the laryngeal vibration signal includes: a vocal cord vibration signal related to phonation, and a vibration signal generated when the neck muscles contract; The lip moving point data includes: movement data of the lips and the face around the lips when speaking.

3. The speech decoding and recognition method of multimodal laryngeal vibration signal and lip motion point data according to claim 1, characterized in that: In step 2, the preprocessing includes signal amplification, filtering and Fourier transform.

4. The method for speech decoding and recognition of multimodal laryngeal vibration signals and lip motion point data according to claim 1, characterized in that: In step 3, the laryngeal vibration signal recognition model includes: a time-frequency domain feature extraction module and a support vector machine classification module connected in sequence; The time-frequency domain feature extraction module is used to extract the time-frequency domain features of the laryngeal vibration signal; The support vector machine classification module is used to process the laryngeal vibration signal and output the tone recognition result of the laryngeal vibration signal.

5. The method for speech decoding and recognition of multimodal laryngeal vibration signals and lip motion point data according to claim 1, characterized in that: In step 3, the time-frequency domain characteristics of the laryngeal vibration signal include the number of zero crossings.

6. The method for speech decoding and recognition of multimodal laryngeal vibration signals and lip motion point data according to claim 1, characterized in that: In step 5, the multimodal fusion model is a TFN model, a LMF model or a PTP model.

7. A speech decoding and recognition system for multimodal laryngeal vibration signals and lip motion point data for implementing the speech decoding and recognition method according to any one of claims 1 to 6, characterized in that: include: A laryngeal vibration signal acquisition module, a laryngeal vibration signal preprocessing module, and a laryngeal vibration signal recognition model connected in sequence; A depth camera, a lip motion point data preprocessing module, and a lip motion point data recognition model connected in sequence; A multimodal fusion model, wherein the input end of the multimodal fusion model is connected to the output end of the laryngeal vibration signal recognition model and the output end of the lip moving point data recognition model; A large language model connected to the output of the multimodal fusion model; The laryngeal vibration signal acquisition module is used to acquire laryngeal vibration signals generated by vocal cord vibration and neck muscle contraction related to phonation, the laryngeal vibration signal preprocessing module is used to preprocess the acquired laryngeal vibration signals, and the laryngeal vibration signal recognition model is used to perform feature extraction, tone and Chinese four tones classification and speech prediction on the laryngeal vibration signals of the preprocessed larynx; The depth camera is used to collect the movement data of the lips and the face around the lips when speaking, the lip moving point data preprocessing module is used to preprocess the collected lip moving point data, and the lip moving point data recognition model is used to extract features from the preprocessed moving point data and recognize syllables; The multimodal fusion model synthesizes the tone results recognized by the laryngeal vibration recognition model and the syllable results recognized by the lip motion data model, outputs the synthesized results, sends them to the large language model, and outputs the final recognition results.

8. The speech decoding and recognition system according to claim 7, characterized in that: The laryngeal vibration signal acquisition module comprises one or more signal acquisition channels, and each acquisition channel comprises a piezoelectric sensor for acquiring laryngeal vibration signals.

9. The speech decoding and recognition system according to claim 7, characterized in that: The laryngeal vibration signal preprocessing module comprises a signal amplifier, a filter and a Fourier transformer.

Images