A multi-modal lip reading method and device fusing respiratory airflow data

Abstract

The application discloses a kind of multi-modal lip reading method and device of fusion respiratory airflow data, method includes: obtaining pronunciation action video signal and respiratory airflow signal;The first pre-processing is carried out to pronunciation action video signal, and pronunciation action video is obtained;Respiratory airflow signal is secondly pre-processed, and respiratory airflow data is obtained;Pronunciation action video is carried out visual feature extraction processing, and lip movement feature is obtained;Respiratory airflow data is carried out component analysis processing, and frequency component and intensity component are obtained;According to frequency component and intensity component, airflow feature extraction processing is carried out to respiratory airflow data, and airflow feature is obtained;Lip movement feature and airflow feature are carried out multi-modal fusion processing, and fusion feature is obtained;Fusion feature is input into lip reading model, and lip reading result is obtained.The application realizes multi-modal lip reading, improves accuracy and applicability.The application can be widely applied in artificial intelligence visual speech recognition technical field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence visual speech recognition, and in particular to a multi-modal lip speech recognition method and device fusing respiratory airflow data. BACKGROUND

[0002] Laryngectomy is an effective way to cure laryngeal cancer, but it will cause the patient to lose the ability to produce normal voice. Traditional solutions (such as writing boards, sign language, electronic larynx or esophageal speech, etc.) are still not convenient and intuitive in daily communication, and have limitations. The visual speech recognition program (VSRP) can convert the soundless pronunciation action into the expected speech text, and then broadcast it through the loudspeaker. However, the existing model training data set is mostly from standard pronunciation healthy people, and the lip movement of healthy people is synchronized with the sound, while the lip movement and expression of patients after total laryngectomy are often more exaggerated when they speak, and the model training data set usually only has a single modality of facial images, resulting in low recognition accuracy and low applicability of the model.

[0003] In summary, the technical problems in the related art need to be improved. SUMMARY

[0004] The present application provides a multi-modal lip speech recognition method and device fusing respiratory airflow data, which effectively improves the accuracy and applicability.

[0005] In one aspect, the present application provides a multi-modal lip speech recognition method fusing respiratory airflow data, comprising the following steps:

[0006] Obtaining a pronunciation action video signal and a respiratory airflow signal;

[0007] Firstly, the pronunciation action video signal is pre-processed to obtain a pronunciation action video;

[0008] Secondly, the respiratory airflow signal is pre-processed to obtain respiratory airflow data;

[0009] Thirdly, visual feature extraction processing is performed on the pronunciation action video to obtain lip movement features;

[0010] Fourthly, component analysis processing is performed on the respiratory airflow data to obtain frequency components and intensity components;

[0011] Fifthly, airflow feature extraction processing is performed on the respiratory airflow data according to the frequency components and the intensity components to obtain airflow features, wherein the airflow features include frequency features, intensity features or time features;

[0012] Sixthly, multi-modal fusion processing is performed on the lip movement features and the airflow features to obtain fusion features;

[0013] input the fusion feature into a lip speech recognition model to obtain a lip speech recognition result.

[0014] In some embodiments, the first preprocessing of the pronunciation action video signal to obtain a pronunciation action video comprises:

[0015] performing digital signal conversion processing on the pronunciation action video signal to obtain a first video;

[0016] performing frame extraction processing on the first video to obtain a second video;

[0017] performing time sequence alignment processing on the second video to obtain a third video;

[0018] performing grayscale processing on the third video to obtain a fourth video;

[0019] performing background noise elimination processing on the fourth video by using a Gaussian background model to obtain the pronunciation action video.

[0020] In some embodiments, the second preprocessing of the respiratory airflow signal to obtain respiratory airflow data comprises:

[0021] performing digital signal conversion processing on the respiratory airflow signal to obtain first airflow data;

[0022] performing denoising processing on the first airflow data by using a preset denoising method to obtain second airflow data, the preset denoising method comprising wavelet transform or Fourier transform;

[0023] performing filtering processing on the second airflow data by using a preset filter to obtain third airflow data, the preset filter comprising a low-pass filter or a band-pass filter;

[0024] performing amplification processing on the third airflow data to obtain the respiratory airflow data.

[0025] In some embodiments, the visual feature extraction processing on the pronunciation action video to obtain lip movement features comprises:

[0026] performing lip region positioning on the pronunciation action video by using a face detection method to obtain a lip video;

[0027] performing motion trajectory extraction on the lip video by using an optical flow method to obtain a motion trajectory;

[0028] performing motion feature extraction on the motion trajectory by using a preset deep learning network to obtain the lip movement features.

[0029] In some embodiments, the component analysis processing on the respiratory airflow data to obtain the frequency component and the intensity component comprises:

[0030] The frequency analysis on the respiratory airflow data by using the frequency domain analysis method to obtain the frequency component;

[0031] The intensity analysis on the respiratory airflow data by using the time domain analysis method to obtain the intensity component.

[0032] In some embodiments, the airflow feature extraction processing on the respiratory airflow data according to the frequency component and the intensity component to obtain the airflow feature comprises:

[0033] The frequency feature extraction on the respiratory airflow data according to the frequency component by using the autocorrelation function or the power spectral density to obtain the frequency feature, the frequency feature comprising the fundamental frequency, the frequency variation range or the frequency stability;

[0034] The intensity feature extraction on the respiratory airflow data according to the intensity component by using the statistical analysis method to obtain the intensity feature, the intensity feature comprising the airflow peak value, the average airflow intensity or the airflow variation rate, the statistical analysis method comprising the mean method, the variance method or the peak method;

[0035] The time feature extraction on the respiratory airflow data by using the time series analysis method to obtain the time feature, the time feature comprising the airflow duration or the airflow variation time sequence, the time series analysis method comprising the difference method or the integral method.

[0036] In some embodiments, the multi-modal fusion processing on the lip movement feature and the airflow feature to obtain the fusion feature comprises:

[0037] The time alignment on the lip movement feature and the airflow feature according to the time stamp by using the time alignment method, the time alignment method comprising the dynamic time warping;

[0038] The query matrix is generated according to the time-aligned airflow feature;

[0039] The key matrix and the value matrix are generated according to the time-aligned lip movement feature;

[0040] The attention weight is calculated according to the query matrix, the key matrix, the value matrix, the dimension of the key and the preset activation function;

[0041] The feature fusion on the time-aligned lip movement feature and the airflow feature is performed according to the attention weight to obtain the fusion feature.

[0042] In some embodiments, the lip speech recognition model is obtained by the following steps:

[0043] obtain a laryngectomee pronunciation action video dataset;

[0044] label the laryngectomee pronunciation action video dataset to obtain a training set;

[0045] According to the performance evaluation index, the training set is input into an initial deep learning model to train the initial deep learning model to obtain the labial speech recognition model.

[0046] In some embodiments, the construction process of the initial deep learning model includes:

[0047] An image feature encoding module is constructed, which is used to generate a feature sequence by using a deep residual network according to a video frame sequence;

[0048] After the image feature encoding module, a time sequence feature extraction module is constructed, which is used to generate a time sequence feature by using a multi-head attention mechanism according to the feature sequence.

[0049] In another aspect, the embodiments of the present application provide a multi-modal labial speech recognition device fusing respiratory airflow data, comprising:

[0050] A first module is used to obtain pronunciation action video signals and respiratory airflow signals;

[0051] A second module is used to perform first preprocessing on the pronunciation action video signals to obtain pronunciation action videos;

[0052] A third module is used to perform second preprocessing on the respiratory airflow signals to obtain respiratory airflow data;

[0053] A fourth module is used to perform visual feature extraction processing on the pronunciation action videos to obtain lip movement features;

[0054] A fifth module is used to perform component analysis processing on the respiratory airflow data to obtain frequency components and intensity components;

[0055] A sixth module is used to perform airflow feature extraction processing on the respiratory airflow data according to the frequency components and the intensity components to obtain airflow features, wherein the airflow features include frequency features, intensity features or time features;

[0056] A seventh module is used to perform multi-modal fusion processing on the lip movement features and the airflow features to obtain fusion features;

[0057] An eighth module is used to input the fusion features into a labial speech recognition model to obtain a labial speech recognition result.

[0058] The application has the following beneficial effects:

[0059] The embodiment of the application first acquires a pronunciation action video signal and a breathing airflow signal, performs first preprocessing on the pronunciation action video signal to obtain a pronunciation action video, and performs second preprocessing on the breathing airflow signal to obtain breathing airflow data, then performs visual feature extraction processing on the pronunciation action video to obtain a lip movement feature, performs component analysis processing on the breathing airflow data to obtain a frequency component and an intensity component, and performs airflow feature extraction processing on the breathing airflow data according to the frequency component and the intensity component to obtain an airflow feature, then performs multi-modal fusion processing on the lip movement feature and the airflow feature to obtain a fusion feature, and finally inputs the fusion feature into a lip language recognition model to obtain a lip language recognition result, so that multi-modal lip language recognition can be realized through the lip movement feature and the airflow feature, and the accuracy and applicability are improved.

[0060] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application can be achieved and obtained by means of the structure particularly pointed out in the description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0062] Figure 1 A flow chart of a multi-modal lip language recognition method fusing breathing airflow data according to an embodiment of the present application;

[0063] Figure 2 A schematic diagram of multi-modal data cross fusion combined with an attention mechanism according to an embodiment of the present application;

[0064] Figure 3 A schematic diagram of an initial deep learning model architecture according to an embodiment of the present application;

[0065] Figure 4 A structural schematic diagram of a multi-modal lip language recognition device fusing breathing airflow data according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are intended to explain, not to limit, the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with embodiments of the present application. They are merely examples of apparatuses and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0067] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0068] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding plurality, and any refers to any one of the plurality.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0070] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are explained, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0071] Lipreading: Also known as Lip Reading or Visual Speech Recognition, it refers to the process of understanding and recognizing the speech content of a speaker by observing their lip movements, facial expressions, and movements of the vocal organs. Artificial intelligence visual speech recognition technology is a combination of computer vision and speech recognition technology. Computer vision enables computers to understand visual information from images or videos, while speech recognition technology enables computers to recognize and process human speech. This technology can simulate human perception, enabling computers to simultaneously understand and analyze visual and auditory data, thereby improving the accuracy and efficiency of information processing. This technology can be used to assist patients after laryngectomy in speech reconstruction. Lipreading is a computer vision technology that can recognize the words spoken by patients after laryngectomy by analyzing their lip movements and generate text, and output sound through speech synthesis technology, to achieve daily communication and exchange for laryngectomees. Through the software application installed and running on the smartphone. In this case, the phone application may integrate lipreading technology, making it a portable aid for laryngectomees to communicate daily.

[0072] Total Laryngectomy: A surgical procedure to treat laryngeal cancer or other laryngeal diseases, which involves the removal of the entire laryngeal structure. After surgery, patients lose the ability to produce sound through traditional means such as vocal cord vibration, becoming laryngectomees.

[0073] In the related art, laryngectomy is an effective way to cure laryngeal cancer, but it causes the patient to lose the ability to normally produce sound and severely limits daily communication. Loss of voice can significantly reduce the patient's quality of life, as being unable to normally speak and communicate can cause great pain, anxiety, fear, distress and frustration. In a survey of patients who were mechanically ventilated after laryngectomy, 82% of the patients indicated that they were moderately to extremely bothered by not being able to speak. Therefore, the need for voice restoration is urgent for these patients. Traditional solutions such as a writing board, sign language, an electronic larynx, and esophageal speech are not convenient and intuitive for daily communication, and all have their limitations. Tracheoesophageal speech, although the "gold standard", has the most complications; esophageal speech and electronic larynx speech, although more natural than tracheoesophageal speech, still have problems of speech distortion, weakening and difficulty in recognition in noisy environments or when using the phone. In addition, these methods cannot be used immediately after surgery and require waiting for the surgical sutures to fully heal. In order to overcome these limitations, researchers have made efforts to develop visual-based speech recognition programs, namely visual speech recognition programs (VSRP). Such programs can convert silent articulatory movements into expected speech text, which is then broadcast through a loudspeaker. VSRP is essentially a computer-aided lip-reading program that relies on hardware to capture articulatory movement data, which is then converted into speech text. More importantly, existing lip-reading recognition datasets are mostly derived from standard articulatory healthy people, and the lip movements of healthy people are synchronized with the sound. After total laryngectomy, the larynx of the patient without a larynx is removed, and the lip movements and expressions during speech are often more exaggerated, and the amplitude of the lip, mandibular expression muscle movements during articulation may be different from that of normal people. The accuracy and applicability of the lip speech recognition model trained using the lip speech dataset of healthy people may be greatly reduced. First, the person without a larynx usually communicates through an artificial larynx or esophageal speech. This way of speaking may cause the lip movements to be less clear than normal, or the amplitude of the lip movements or facial expressions to be greater than normal; second, the lip feature of the person without a larynx may change due to the absence of the vocal cords, and the lip feature of some syllables may not be obvious or typical. Third, due to the absence of the larynx, visual cues such as laryngeal vibration are missing, and only lip movements and facial expressions can be relied on for lip speech recognition, which may require stronger data acquisition and feature extraction capabilities. The above factors increase the difficulty of lip speech recognition, making lip speech recognition for patients without a larynx relatively more challenging. In related research, the peak expiratory flow and peak air pressure during breathing of laryngectomees are higher than those of normal people. Laryngectomees exhibit three different breathing patterns when speaking with an electronic larynx: breath holding, expiration, and inspiration. Four of the 12 laryngectomees who use an electronic larynx for a long time hold their breath when speaking, seven of them continuously exhale when speaking, and only one of them maintains breathing when speaking. Therefore, the aerodynamic characteristics such as breathing frequency and intensity of laryngectomees are quite different from those of normal people, and the changes in breathing frequency and intensity during speaking will affect the accuracy of lip speech recognition.The lip movement, facial muscle movement, airflow frequency and intensity of the laryngectomy patient when speaking are different from those of healthy people, and the two features are fused into the trained lip-reading model, so as to improve the specificity and applicability of the model in the laryngectomy group.

[0074] Therefore, the embodiment extracts the frequency and intensity of the tracheostomy breathing airflow of the laryngectomee and the features of the lip-reading video, performs multi-modal fusion, and then performs lip-reading, which can effectively improve the convenience and intuitiveness of the laryngectomee in daily communication, and improve the accuracy and applicability of the recognition.

[0075] The multi-modal lip-reading method fusing breathing airflow data provided by the embodiment of the application relates to the technical field of artificial intelligence visual speech recognition. The multi-modal lip-reading method fusing breathing airflow data provided by the embodiment of the application can be applied to a terminal, can also be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application for implementing a multi-modal lip-reading method fusing breathing airflow data, and the like, but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0077] The embodiments of the present application will be specifically explained below in combination with the drawings:

[0078] Figure 1 is an optional flowchart of a multi-modal lip reading method fusing breath airflow data provided by an embodiment of the present application, Figure 1 The method in the method can include but is not limited to steps S101-S108.

[0079] Step S101, obtaining pronunciation action video signals and breath airflow signals;

[0080] Step S102, first preprocessing the pronunciation action video signals to obtain pronunciation action videos;

[0081] Step S103, second preprocessing the breath airflow signals to obtain breath airflow data;

[0082] Step S104, visual feature extraction processing on the pronunciation action videos to obtain lip movement features;

[0083] Step S105, component analysis processing on the breath airflow data to obtain frequency components and intensity components;

[0084] Step S106, airflow feature extraction processing on the breath airflow data according to the frequency components and the intensity components to obtain airflow features, the airflow features including frequency features, intensity features, or time features;

[0085] Step S107, multi-modal fusion processing on the lip movement features and the airflow features to obtain fusion features;

[0086] Step S108, inputting the fusion features into a lip reading model to obtain a lip reading result.

[0087] The steps S101-S108 shown in the embodiments of the present application realize multi-modal lip reading, and improve the accuracy and applicability.

[0088] In step S101 of some embodiments, the pronunciation action video signal can be acquired by a high-definition camera, and the respiratory airflow signal can be acquired by an airflow sensing device. The pronunciation action video signal and the respiratory airflow signal can also be acquired by other means, which are not limited. Illustratively, in the pronunciation action video signal acquisition, a high-definition camera with a frame rate of at least 30 fps can be used to capture the subtle changes in lip, jaw movement and facial expression. The optimal installation position of the high-definition camera is to face the patient's face at a distance of about 40-50 cm to ensure that the details of the lip, jaw and facial expression can be clearly captured. In the respiratory airflow signal acquisition, a high-sensitivity micro airflow sensor can be used to accurately capture the airflow changes at the air cut. A sensor such as a thermistor or differential pressure sensor that responds quickly and accurately to airflow changes can be used. The sensor is embedded in the air cut mask or neck cover worn by the laryngectomee to ensure that the sensor is in close contact with the air cut and can accurately measure the airflow data.

[0089] In some embodiments, the first preprocessing of the pronunciation action video signal in step S102 to obtain the pronunciation action video can include but is not limited to the following steps:

[0090] Digital signal conversion processing is performed on the pronunciation action video signal to obtain a first video;

[0091] Frame extraction processing is performed on the first video to obtain a second video;

[0092] Time sequence alignment processing is performed on the second video to obtain a third video;

[0093] Gray scale processing is performed on the third video to obtain a fourth video;

[0094] The fourth video is subjected to background noise elimination processing using a Gaussian background model to obtain the pronunciation action video.

[0095] In some embodiments, the pronunciation action video signal can be first converted into a digital signal by the image processing unit to obtain a first video, so that the video signal is converted into a digital signal. Then the first video is frame-extracted to obtain a second video. Exemplarily, since the number of video frames is large, and there is a lot of redundant information between frames (the adjacent frames of a certain frame are actually the same as the content expressed by the frame), a large number of redundant frames will only increase the duration of model training, and the video frames can be processed by frame-extraction technology, that is, every several frames are extracted as data, and the redundant frames are deleted. The second video is then time-series aligned to obtain a third video. Exemplarily, since the frame-extraction technology is used, the obtained video frames are not the original time sequence, and the time sequence can be adjusted according to the frame-extraction ratio, and each video frame is time-aligned to make each aligned frame have the same time length and resolution. Then the third video is grayed to obtain a fourth video. It can be understood that converting a color image into a gray image can reduce the data amount and highlight the facial features. Finally, the fourth video is processed by a Gaussian background model to eliminate background noise to obtain a pronunciation action video, so as to eliminate background noise and only keep the pronunciation action of the laryngectomee. More, a deep learning algorithm can also be used for background noise elimination processing.

[0096] In some embodiments, the second preprocessing of the respiratory airflow signal in step S103 to obtain the respiratory airflow data can include but is not limited to the following steps:

[0097] The respiratory airflow signal is converted into a digital signal to obtain first airflow data;

[0098] The first airflow data is denoised by a preset denoising method to obtain second airflow data, and the preset denoising method includes wavelet transform or Fourier transform;

[0099] The second airflow data is filtered by a preset filter to obtain third airflow data, and the preset filter includes a low-pass filter or a band-pass filter;

[0100] The third airflow data is amplified to obtain the respiratory airflow data.

[0101] In some embodiments, the respiratory airflow signal can be first processed by the signal processing unit to obtain first airflow data, so that the airflow signal is converted into a digital signal. Then, the first airflow data is denoised using a preset denoising method to obtain second airflow data, wherein the preset denoising method includes wavelet transform or Fourier transform. For example, the wavelet transform or Fourier transform and other signal processing techniques can be used to remove the noise components in the airflow signal (first airflow data) and retain the effective signal to obtain the second airflow data. Then, the second airflow data is filtered using a preset filter to obtain third airflow data, wherein the preset filter includes a low-pass filter or a band-pass filter. For example, the low-pass filter or the band-pass filter can be used to filter out the high-frequency noise and low-frequency drift in the second airflow data, and only retain the signal components related to the airflow frequency to obtain the third airflow data. Finally, the third airflow data is amplified to obtain the respiratory airflow data, so that the amplitude of the airflow signal in the third airflow data has sufficient dynamic range in subsequent processing.

[0102] In some embodiments, in step S104, the visual feature extraction processing of the pronunciation action video is performed to obtain the lip movement feature, which can include but is not limited to the following steps:

[0103] The lip region of the pronunciation action video is located using a face detection method to obtain a lip video;

[0104] The motion trajectory of the lip video is extracted using an optical flow method to obtain a motion trajectory;

[0105] The motion trajectory is extracted using a preset deep learning network to obtain a lip movement feature.

[0106] In some embodiments, the lip region of the pronunciation action video can be first located using a face detection method to obtain a lip video. For example, the face detection algorithm (Haar Cascades) of OpenCV can be used to locate the lip region, and the lip video can be obtained by cropping the lip region. Then, the motion trajectory of the lip video is extracted using an optical flow method. The motion trajectory of the pixels in the image is calculated by analyzing the pixel changes in the image sequence. Finally, the motion trajectory is extracted using a preset deep learning network (such as a convolutional neural network, a recurrent neural network, a Transformer, etc.) to extract the motion features of the lips, the lower jaw, and the facial expression, thereby obtaining the lip movement feature.

[0107] In some embodiments, in step S105, the respiratory airflow data is analyzed to obtain the frequency component and the intensity component, which can include but is not limited to the following steps:

[0108] The frequency analysis of the respiratory airflow data is performed using a frequency domain analysis method to obtain a frequency component;

[0109] The intensity analysis is performed on the respiratory airflow data by using the time domain analysis method to obtain the intensity component.

[0110] In some embodiments, the frequency analysis is performed on the respiratory airflow data by using the frequency domain analysis method to obtain the frequency component, and the frequency component in the respiratory airflow data is extracted by using the frequency domain analysis method (such as fast Fourier transform (FFT)) to analyze the fundamental frequency and harmonic component of the airflow and identify the fundamental frequency and frequency change during pronunciation. Then, the intensity analysis is performed on the respiratory airflow data by using the time domain analysis method to obtain the intensity component, and the intensity component in the respiratory airflow data is extracted by using the time domain analysis method (such as peak detection, average calculation, etc.) to analyze the amplitude change of the signal in the airflow and identify the intensity change of the airflow.

[0111] In some embodiments, in step S106, the airflow feature extraction processing is performed on the respiratory airflow data according to the frequency component and the intensity component to obtain the airflow feature, which can include but is not limited to the following steps:

[0112] According to the frequency component, the frequency feature extraction is performed on the respiratory airflow data by using the autocorrelation function or the power spectral density to obtain the frequency feature, and the frequency feature includes the fundamental frequency, the frequency change range or the frequency stability;

[0113] According to the intensity component, the intensity feature extraction is performed on the respiratory airflow data by using the statistical analysis method to obtain the intensity feature, and the intensity feature includes the airflow peak value, the average airflow intensity or the airflow change rate, and the statistical analysis method includes the mean method, the variance method or the peak method;

[0114] The time feature extraction is performed on the respiratory airflow data by using the time series analysis method to obtain the time feature, and the time feature includes the airflow duration or the airflow change time sequence, and the time series analysis method includes the difference method or the integral method.

[0115] In some embodiments, airflow feature extraction processing can be performed on respiratory airflow data based on frequency and intensity components to obtain airflow features, which include frequency features, intensity features, or time features. In frequency feature extraction, frequency features can be extracted from respiratory airflow data based on frequency components using autocorrelation functions or power spectral density, where frequency features may include the fundamental frequency, frequency variation range, or frequency stability. In intensity feature extraction, intensity features can be extracted from respiratory airflow data based on intensity components using statistical analysis methods, where intensity features may include peak airflow, average airflow intensity, or airflow change rate, and statistical analysis methods may include the mean method, variance method, or peak value method. In time feature extraction, time series analysis methods can be used to extract time features from respiratory airflow data, where time features may include airflow duration or airflow change sequence, and time series analysis methods may include the difference method or the integral method.

[0116] In some embodiments, step S107 involves multimodal fusion processing of the lip movement features and airflow features to obtain fused features, which may include, but is not limited to, the following steps:

[0117] Based on the timestamp, the lip movement features and airflow features are time-aligned using a time alignment method, which includes dynamic time warping.

[0118] Generate a query matrix based on the time-aligned airflow characteristics;

[0119] Generate a key matrix and a value matrix based on the time-aligned lip movement features;

[0120] Calculate attention weights based on the query matrix, key matrix, value matrix, key dimensions, and preset activation function;

[0121] Based on attention weights, the time-aligned lip movement features and airflow features are fused to obtain fused features.

[0122] In some embodiments, lip movement features and airflow features can be time-aligned based on timestamps using a time alignment method. This time alignment method may include dynamic time warping to synchronize lip movement features and airflow features along the time axis. Multimodal data cross-fusion combined with attention mechanisms, such as... Figure 2 As shown, a query matrix Q can be generated based on the time-aligned airflow features, and a key matrix K and a value matrix V can be generated based on the time-aligned lip movement features. Then, attention weights are calculated based on the dimensions of the query matrix, key matrix, value matrix, and keys, and a preset activation function. The formula for calculating the attention weights is as follows: In the formula, Attention is an attention weight, Q is a query matrix, K is a key matrix, V is a value matrix, Softmax is an activation function, d k is the dimension of the key. Finally, according to the attention weight, the time-aligned lip movement feature and the airflow feature are fused to obtain a fused feature. It can be understood that the airflow feature can be taken as the query matrix Q, and the lip movement feature can be taken as the key matrix K and the value matrix V, so as to model the relationship between the two modalities and fully fuse the airflow feature and the lip movement feature.

[0123] In some embodiments, in step S108, the fused feature can be input into a lip speech recognition model to obtain a lip speech recognition result. For example, the fused feature can be input into a trained lip speech recognition model, and the final lip speech recognition result can be output through a neural network, a model front-end encoder and a decoder.

[0124] In some embodiments, the lip speech recognition model is obtained through the following steps:

[0125] Obtain a laryngectomee pronunciation action video dataset;

[0126] Label the laryngectomee pronunciation action video dataset to obtain a training set;

[0127] According to the performance evaluation index, input the training set into an initial deep learning model to train the initial deep learning model to obtain a lip speech recognition model.

[0128] In some embodiments, a laryngectomee pronunciation action video dataset can be obtained first, which contains video samples under different pronunciation actions, different lighting conditions and different background environments. Then, the laryngectomee pronunciation action video dataset is labeled to obtain a training set. Then, according to the performance evaluation index, the training set is input into an initial deep learning model to train the initial deep learning model to obtain a lip speech recognition model. During the training process, data augmentation techniques such as image rotation, scaling and color transformation can be used to increase the diversity of the dataset and improve the generalization ability of the model. After the training is completed, an independent test dataset can be used to evaluate the performance of the trained model, and the accuracy, recall rate, F1 score and other indicators of the model can be calculated to ensure that the performance of the model meets the requirements. More specifically, the present embodiment can efficiently process video data in real time, capture and analyze real-time pronunciation actions of laryngectomees, and has the characteristics of low latency and high throughput, which can complete action capture and feature extraction within milliseconds to realize real-time processing. The present embodiment has a real-time feedback mechanism, which can output speech recognition results in real time according to the pronunciation actions of laryngectomees. The feedback mechanism can be realized through speech synthesis, text display and other ways to ensure that laryngectomees can obtain feedback in a timely manner.

[0129] In some embodiments, the construction process of the initial deep learning model includes:

[0130] constructing an image feature encoding module, the image feature encoding module being configured to generate a feature sequence from the video frame sequence using a deep residual network;

[0131] after the image feature encoding module, constructing a temporal feature extraction module, the temporal feature extraction module being configured to generate a temporal feature from the feature sequence using a multi-head attention mechanism.

[0132] In some embodiments, the initial deep learning model architecture is as shown in Figure 3 the image feature encoding module can be constructed first, and then the temporal feature extraction module can be constructed. The image feature encoding module is configured to generate a feature sequence from the video frame sequence using a deep residual network. It can be understood that in the image feature encoding module, in order to enhance the extraction ability of the lip feature, a deep ResNet (deep residual network) model that has been pre-trained on a large-scale lip dataset can be used as an image feature encoder, which combines a multi-level convolution feature extraction mechanism, can not only capture fine-grained features of the lips, but also effectively suppress information loss and gradient disappearance problems by introducing a residual learning structure while preserving important details. Further, the temporal feature extraction module is configured to generate a temporal feature from the feature sequence using a multi-head attention mechanism. It can be understood that since lip movement is a dynamic process with a time dimension, a temporal feature extraction module can be introduced. The traditional LSTM structure is relatively fixed for processing time series, while the multi-head attention mechanism (Multi-Head Attention) is used in this embodiment, which enables adaptive adjustment of the window of temporal modeling according to the lip movement feature at each time, dynamically captures key temporal information, and obtains the temporal feature. Furthermore, in order to further improve the adaptability to complex lip movements, image style conversion technology can be combined to enable the model to automatically enhance the contrast and edge sharpness of the lip region, so that efficient feature extraction can still be maintained under complex background and lighting conditions, to achieve more robust and accurate lip feature coding, effectively improving the performance of lip reading.

[0133] In some embodiments, the present embodiment captures the lip-reading data and airflow data of laryngectomee patients through high-definition cameras and miniature airflow sensors, respectively, uses advanced image processing and signal processing techniques to accurately analyze lip movements, amplitudes, jaw movements, facial expressions, and airflow frequencies and intensities, and fuses the lip-reading data and airflow data through multi-modal data fusion technology, significantly improving the accuracy of laryngectomee speech recognition, enabling the system to more accurately recognize the speech information of laryngectomee patients. The present embodiment uses a high-sensitivity miniature airflow sensor to monitor the airflow frequency and intensity changes at the tracheostomy in real time and accurately analyzes the time, frequency, and intensity characteristics of the airflow through signal processing techniques. By accurately capturing airflow characteristics, the pronunciation method of laryngectomee patients can be more comprehensively understood, enhancing the robustness and accuracy of the speech recognition system. The present embodiment uses multi-modal data fusion technology to time-align and weight fuse the lip-reading data and airflow data, generating comprehensive multi-modal fusion features as important inputs for speech recognition. Multi-modal data fusion technology can comprehensively utilize visual and airflow information, enhancing the system's feature expression ability, improving the accuracy and reliability of speech recognition, and reducing misrecognition rates. The present embodiment uses efficient image processing and signal processing techniques to ensure that the system can efficiently process video and airflow data in real-time environments, achieving millisecond-level low latency. Real-time processing and low latency characteristics enable the system to timely feedback the pronunciation results of laryngectomee patients, improving the naturalness and fluency of interaction. Therefore, the present embodiment significantly improves the accuracy of laryngectomee speech recognition and the robustness of the system through high-precision data capture, advanced image processing and signal processing techniques, multi-modal data fusion, and the application of deep learning models, with real-time and low latency characteristics, strong adaptability, and high universality. Not only does it enhance the convenience and intuitiveness of laryngectomee patients in daily communication, but also improves their quality of life, having important social significance and application value.

[0134] The beneficial effects of implementing the embodiments of the present application include: the embodiments of the present application first acquire pronunciation action video signals and respiratory airflow signals, perform first preprocessing on the pronunciation action video signals to obtain pronunciation action videos, and perform second preprocessing on the respiratory airflow signals to obtain respiratory airflow data, then perform visual feature extraction processing on the pronunciation action videos to obtain lip movement features, perform component analysis processing on the respiratory airflow data to obtain frequency components and intensity components, and perform airflow feature extraction processing on the respiratory airflow data according to the frequency components and the intensity components to obtain airflow features, then perform multi-modal fusion processing on the lip movement features and the airflow features to obtain fusion features, and finally input the fusion features into a lip-reading model to obtain a lip-reading result, so that multi-modal lip-reading can be realized through the lip movement features and the airflow features, thereby improving the accuracy and applicability.

[0135] As Figure 4As shown, the embodiment of the present application further provides a multi-modal lip speech recognition device fusing breath airflow data, comprising:

[0136] The first module 801 is configured to acquire the pronunciation action video signal and the breath airflow signal.

[0137] The second module 802 is configured to perform first preprocessing on the pronunciation action video signal to obtain the pronunciation action video.

[0138] The third module 803 is configured to perform second preprocessing on the breath airflow signal to obtain the breath airflow data.

[0139] The fourth module 804 is configured to perform visual feature extraction processing on the pronunciation action video to obtain the lip movement feature.

[0140] The fifth module 805 is configured to perform component analysis processing on the breath airflow data to obtain the frequency component and the intensity component.

[0141] The sixth module 806 is configured to perform airflow feature extraction processing on the breath airflow data according to the frequency component and the intensity component to obtain the airflow feature, wherein the airflow feature comprises the frequency feature, the intensity feature or the time feature.

[0142] The seventh module 807 is configured to perform multi-modal fusion processing on the lip movement feature and the airflow feature to obtain the fusion feature.

[0143] The eighth module 808 is configured to input the fusion feature into the lip speech recognition model to obtain the lip speech recognition result.

[0144] The content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0145] The preferred embodiments of the present application are described above with reference to the accompanying drawings, and the scope of the present application is not limited by this. Any modification, equivalent replacement and improvement made by those skilled in the art without departing from the scope and essence of the present application shall be within the scope of the present application.

Claims

1. A multimodal lip-reading recognition method integrating respiratory airflow data, characterized in that, Includes the following steps: Acquire video signals of vocalization movements and respiratory airflow signals; The vocalization video signal is subjected to a first preprocessing step to obtain the vocalization video. The respiratory airflow signal undergoes a second preprocessing step to obtain respiratory airflow data; Visual feature extraction processing is performed on the video of the pronunciation action to obtain lip movement features; The respiratory airflow data were subjected to component analysis to obtain frequency and intensity components; Based on the frequency component and the intensity component, the respiratory airflow data is subjected to airflow feature extraction processing to obtain airflow features, which include frequency features, intensity features or time features; The lip movement features and the airflow features are subjected to multimodal fusion processing to obtain fused features; The fused features are input into the lip reading recognition model to obtain the lip reading recognition result; The step of extracting airflow features from the respiratory airflow data based on the frequency component and the intensity component to obtain airflow features includes: Based on the frequency components, frequency features are extracted from the respiratory airflow data using autocorrelation function or power spectral density to obtain the frequency features, which include fundamental frequency, frequency variation range or frequency stability. Based on the intensity components, the intensity features of the respiratory airflow data are extracted using statistical analysis methods to obtain the intensity features, which include airflow peak value, average airflow intensity, or airflow change rate. The statistical analysis methods include the mean method, variance method, or peak method. The respiratory airflow data is subjected to time series analysis to extract time features, which include airflow duration or airflow change sequence. The time series analysis method includes difference method or integral method.

2. The method according to claim 1, characterized in that, The first preprocessing of the vocalization video signal to obtain the vocalization video includes: The video signal of the vocalization action is digitally converted to obtain the first video. The first video is subjected to frame extraction to obtain the second video; The second video is subjected to time-series alignment processing to obtain the third video; The third video is converted to grayscale to obtain the fourth video; The fourth video is processed by removing background noise using a Gaussian background model to obtain the video of the pronunciation action.

3. The method according to claim 1, characterized in that, The second preprocessing of the respiratory airflow signal to obtain respiratory airflow data includes: The respiratory airflow signal is digitally converted to obtain the first airflow data; The first airflow data is denoised using a preset denoising method to obtain the second airflow data. The preset denoising method includes wavelet transform or Fourier transform. The second airflow data is filtered using a preset filter to obtain the third airflow data. The preset filter includes a low-pass filter or a band-pass filter. The third airflow data is amplified to obtain the respiratory airflow data.

4. The method according to claim 1, characterized in that, The step of performing visual feature extraction processing on the pronunciation action video to obtain lip movement features includes: The lip region is located in the pronunciation video using facial detection methods to obtain a lip video; The motion trajectory of the lip video was extracted using optical flow method to obtain the motion trajectory. The motion trajectory is extracted using a pre-defined deep learning network to obtain the lip movement features.

5. The method according to claim 1, characterized in that, The component analysis of the respiratory airflow data to obtain frequency and intensity components includes: The respiratory airflow data were analyzed using frequency domain analysis to obtain the frequency components. The intensity components are obtained by performing intensity analysis on the respiratory airflow data using time-domain analysis.

6. The method according to claim 1, characterized in that, The multimodal fusion processing of the lip movement features and the airflow features to obtain fused features includes: Based on the timestamp, the lip movement feature and the airflow feature are time-aligned using a time alignment method, which includes dynamic time warping. A query matrix is ​​generated based on the time-aligned airflow characteristics; Based on the time-aligned lip movement features, generate a key matrix and a value matrix; Calculate attention weights based on the query matrix, the key matrix, the value matrix, the dimension of the keys, and the preset activation function; Based on the attention weight, the time-aligned lip movement features and airflow features are fused to obtain the fused features.

7. The method according to claim 1, characterized in that, The lip-reading recognition model is obtained through the following steps: Obtain a dataset of video recordings of vocalizations by patients without a laryngectomy; The training set was obtained by annotating the video dataset of vocalization actions of the laryngeal patients. Based on the performance evaluation metrics, the training set is input into the initial deep learning model to train the initial deep learning model and obtain the lip reading recognition model.

8. The method according to claim 7, characterized in that, The process of constructing the initial deep learning model includes: An image feature encoding module is constructed, which is used to generate feature sequences based on video frame sequences using a deep residual network; Following the image feature encoding module, a temporal feature extraction module is constructed. The temporal feature extraction module is used to generate temporal features based on the feature sequence using a multi-head attention mechanism.

9. A multimodal lip-reading recognition device that integrates respiratory airflow data, characterized in that, include: The first module is used to acquire video signals of vocalization actions and respiratory airflow signals; The second module is used to perform a first preprocessing on the pronunciation action video signal to obtain the pronunciation action video; The third module is used to perform a second preprocessing on the respiratory airflow signal to obtain respiratory airflow data; The fourth module is used to perform visual feature extraction processing on the pronunciation action video to obtain lip movement features; The fifth module is used to perform component analysis on the respiratory airflow data to obtain frequency and intensity components; The sixth module is used to perform airflow feature extraction processing on the respiratory airflow data based on the frequency component and the intensity component to obtain airflow features, which include frequency features, intensity features or time features; The seventh module is used to perform multimodal fusion processing on the lip movement features and the airflow features to obtain fused features; The eighth module is used to input the fused features into the lip-reading recognition model to obtain the lip-reading recognition result; The step of extracting airflow features from the respiratory airflow data based on the frequency component and the intensity component to obtain airflow features includes: Based on the frequency components, frequency features are extracted from the respiratory airflow data using autocorrelation function or power spectral density to obtain the frequency features, which include fundamental frequency, frequency variation range or frequency stability. Based on the intensity components, the intensity features of the respiratory airflow data are extracted using statistical analysis methods to obtain the intensity features, which include airflow peak value, average airflow intensity, or airflow change rate. The statistical analysis methods include the mean method, variance method, or peak method. The respiratory airflow data is subjected to time series analysis to extract time features, which include airflow duration or airflow change sequence. The time series analysis method includes difference method or integral method.

Images