Lip feature fusion speech recognition method and device

Abstract

The invention provides a lip feature fusion speech recognition method and device, and the method comprises the steps: obtaining lip ultrasonic single-mode data features through an ultrasonic loudspeaker and an ultrasonic microphone; acquiring single-mode data features of the lip image by using a camera; performing fusion processing on the lip ultrasonic single-mode data features and the lip image single-mode data features by using a lip two-source mode feature fusion model to obtain multi-mode voice information; and determining a speech recognition text and a speech recognition synthetic audio based on the multi-modal speech information. Visibly, the method provided by the invention can overcome the recognition defect of the single-mode feature, improves the natural degree, accuracy and robustness of the recognition result in a silent mode or high-noise intervention, and is low in algorithm complexity.

Description

Technical Field

[0001] The present invention belongs to the field of voice interaction, and specifically relates to a lip feature fusion voice recognition method and device. Background Art

[0002] Voice interaction is the most common and natural way of interpersonal communication, and is also a reliable and effective carrier for information transmission.

[0003] In the implementation of voice interaction, the most obvious and mature method is to encode and decode the audio signal generated by the speaker and transmitted through the air to achieve voice recognition. However, in scenarios such as field operations, emergency rescue, military applications, and medical rehabilitation, the traditional air-conducted audio channel cannot fully realize or even completely lose the representation of voice information due to high noise interference, strong concealment requirements, and the health status of the speaker. Therefore, it is difficult to achieve voice information encoding in the above complex scenarios on the transmitter side. We need to seek other information modes released by the speaker when speaking or when there is an intention to speak, to assist or replace the role of air-conducted audio. This type of processing of voice information based on non-air-conducted audio information modes is usually called silent speech recognition (or silent communication).

[0004] The commonly used information modalities in silent speech recognition include lip reading, electroencephalogram, surface muscle electrical signals, electromagnetic arthrography, permanent magnet arthrography and ultrasound imaging, but most of them focus on capturing the biological characteristics of the human body. They generally have shortcomings such as complex perception methods, poor cross-subject performance, and poor signal capture stability. As a result, the speech models built based on these modalities alone have low accuracy and weak generalization ability.

[0005] It has many disadvantages, such as high difficulty in modularization, weak human-machine friendliness, and poor practicality and portability.

[0006] In comparison, lip movement video and ultrasonic echo profile are closer to practical applications. Lip movement video is captured by a camera and uses the changes in the speaker's lip movement when speaking to map speech information. Ultrasonic echo profile is an echo waveform obtained by receiving periodic frequency-varied ultrasonic waves reflected and diffracted by the speaker's lip skin deformation when speaking, and using the echo waveform to perform signal preprocessing based on distance and intensity to obtain a visualized ultrasonic distribution feature map, which constructs lip deformation and realizes the mapping of speech information.

[0007] For lip movement videos, there are obvious disadvantages in representing speech information only through changes in lip shape. First, the problem of homophones (lip shapes) and different meanings cannot be avoided, especially as the length of the target speech content increases, which affects comprehensibility. Second, conditions such as lighting and angle will greatly affect image quality, effective information content and subsequent modeling, affecting mapping accuracy. Third, if speech synthesis from pure lip movement videos is to be derived in the future, the lack of acoustic features such as the speaker's tone, voice and rhythm will affect naturalness.

[0008] For the ultrasonic echo contour of the lips, the deformation of the lips is depicted through the echo waveform of the reflection and diffraction of the ultrasound, which has the disadvantages of low signal resolution, sensitivity to disturbance intervention and poor cross-subject capability. Specifically, due to the limitation of sampling frequency, the accuracy of the minimum skin deformation that can be depicted by the ultrasonic echo contour is currently around 3-4 mm. When faced with subtle changes in lip shape or similar lip shapes, the features extracted by the ultrasonic echo contour cannot be modeled in a fine-grained and high-quality manner, which will lead to a decrease in the accuracy of downstream classification tasks or even failure; secondly, since sound waves are sensitive to environmental media, if additional media such as hands intervene in the transmission path of the sound waves, the reflection and diffraction of the sound waves will be greatly disturbed; thirdly, since the facial contours and lip shapes of different users are different, the ultrasonic echo contours of each user are different. When using this modality, it is necessary to model each user individually to obtain the ideal recognition effect. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a lip feature fusion speech recognition method and device, which is beneficial to improving the naturalness, accuracy and robustness of the recognition results in silent mode or high noise intervention, and the algorithm complexity is low.

[0010] In order to solve the above technical problems, the first aspect of the embodiments of the present invention discloses a lip feature fusion speech recognition method, the method comprising:

[0011] S1, using a preset ultrasonic speaker and ultrasonic microphone to obtain lip ultrasonic single-modal data features;

[0012] S2, using a preset camera to obtain lip image single-modal data features;

[0013] S3, using a lip two-source modal feature fusion model to fuse the lip ultrasonic single modal data features and the lip image single modal data features to obtain multimodal speech information;

[0014] S4, determining speech recognition text and speech recognition synthesized audio based on the multimodal speech information.

[0015] As an optional implementation, in the first aspect of the embodiment of the present invention, the use of a preset ultrasonic speaker and an ultrasonic microphone to obtain lip ultrasonic single-mode data features includes:

[0016] S11, using an ultrasonic speaker to continuously emit a first frequency modulated continuous wave toward the lip area of ​​the user;

[0017] S12, using an ultrasonic microphone to receive a second frequency modulated continuous wave to obtain the lip ultrasonic echo signal;

[0018] S13, processing the first frequency modulated continuous wave and the lip ultrasonic echo signal to obtain a differential echo contour curve;

[0019] S14, using a first feature extraction model to process the differential echo contour curve to obtain the lip ultrasonic single-mode data feature.

[0020] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the first frequency modulated continuous wave and the lip ultrasonic echo signal to obtain a differential echo contour curve includes:

[0021] S131, initializing the current window number to 0;

[0022] S132, determining a distance deviation signal based on the first frequency modulated continuous wave and the lip ultrasonic echo signal;

[0023] S133, combining the distance deviation signal and the lip ultrasonic echo signal into an echo contour frame;

[0024] S134, adding 1 to the current window number;

[0025] S135, repeating S132 to S134 until the current number of windows reaches N2, and obtaining N2 echo contour frames; N2 is a preset window number threshold; the N2 echo contour frames constitute an echo contour frame sequence;

[0026] S136, sequentially subtracting adjacent echo contour frames in the echo contour frame sequence to obtain a differential echo contour frame sequence; the differential echo contour frame sequence includes differential echo contour frames;

[0027] S137, plotting all the differential echo contour frames in the differential echo contour frame sequence into the same two-dimensional coordinate system to obtain a differential echo contour curve.

[0028] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the method of obtaining the lip image monomodal data features using a preset camera includes:

[0029] S21, using the camera to collect the movement of the lip area to obtain the lip movement video;

[0030] S22, dividing the lip movement video into a lip movement frame sequence;

[0031] S23, using a second feature extraction model to process the lip motion frame sequence to obtain the lip image unimodal data features.

[0032] As an optional implementation, in the first aspect of the embodiment of the present invention, the lip two-source modality feature fusion model includes a first spatiotemporal feature extraction module, a second spatiotemporal feature extraction module, a spatiotemporal feature splicing module, a binary feature fusion module and a fusion feature processing module;

[0033] The first spatiotemporal feature extraction module is connected to the spatiotemporal feature splicing module, and is used to process the lip ultrasonic single-mode data features to obtain a first spatiotemporal feature vector, and send it to the spatiotemporal feature splicing module;

[0034] The second spatiotemporal feature extraction module is connected to the spatiotemporal feature splicing module, and is used to process the lip image unimodal data features to obtain a second spatiotemporal feature vector, and send it to the spatiotemporal feature splicing module;

[0035] The spatiotemporal feature concatenation module is connected to the binary feature fusion module and is used to concatenate the first spatiotemporal feature vector and the second spatiotemporal feature vector to obtain an initial spatiotemporal feature vector and send it to the binary feature fusion module;

[0036] The binary feature fusion module is connected to the fusion feature processing module, and is used to process the initial spatiotemporal feature vector to obtain the lip modality collaborative feature, and send it to the fusion feature processing module;

[0037] The fusion feature processing module is used to process the lip modal collaborative features to obtain multimodal speech information.

[0038] As an optional implementation, in the first aspect of the embodiment of the present invention, determining the speech recognition text and the speech recognition synthesized audio based on the multimodal speech information includes:

[0039] S41, processing the multimodal voice information to obtain a voice recognition text;

[0040] S42, converting the speech recognition text into initial speech audio;

[0041] S43, objectively evaluating the initial speech audio to obtain an objective evaluation result;

[0042] S44, performing a subjective evaluation on the initial speech audio to obtain a subjective evaluation result;

[0043] S45, judging the objective evaluation result and the subjective evaluation result, and obtaining the speech recognition synthesized audio;

[0044] When both the objective evaluation result and the subjective evaluation result are available, determining the speech recognition synthesized audio to be the initial speech audio;

[0045] When the objective evaluation result or the subjective evaluation result is unavailable, it is determined that the speech recognition synthesized audio is empty.

[0046] As an optional implementation manner, in the first aspect of the embodiment of the present invention, objectively evaluating the initial speech audio to obtain an objective evaluation result includes:

[0047] S431, obtaining an objective evaluation value based on the initial speech audio;

[0048] S432, judging the objective evaluation value;

[0049] When the objective evaluation value is greater than or equal to a preset first objective threshold value and less than or equal to a preset second objective threshold value, determining that the objective evaluation result is available;

[0050] When the objective evaluation value is less than the first objective threshold, or greater than the second objective threshold, the objective evaluation result is determined to be unavailable.

[0051] A second aspect of an embodiment of the present invention discloses a lip feature fusion speech recognition device, the device comprising a head-mounted support module, a data acquisition module, a data analysis module and an interaction module;

[0052] The head-mounted support module is worn on the user's head and is used to fix the data acquisition module above the lip area of ​​the user's face, and to fix the data analysis module and the interaction module on the side of the user's head;

[0053] The data acquisition module is used to collect the lip ultrasonic echo signal and the lip movement video, and send them to the data analysis module;

[0054] The data analysis module is data-connected to the interaction module, and is used to fuse the lip ultrasound single-modal data features and the lip image single-modal data features using a lip two-source modal feature fusion model to obtain multi-modal voice information, and send it to the interaction module;

[0055] The interaction module is used to determine the speech recognition text and the speech recognition synthesized audio based on the multimodal speech information.

[0056] As an optional implementation, in the second aspect of the embodiment of the present invention, the data acquisition module includes a camera, an ultrasonic speaker and an ultrasonic microphone;

[0057] The camera is fixed directly above the lip area through the head-mounted bracket module to collect the lip movement video;

[0058] The ultrasonic speaker and the ultrasonic microphone are respectively fixed on both sides above the lip area through the head-mounted bracket module;

[0059] The ultrasonic speaker is used to transmit the first frequency modulated continuous wave to the lip area;

[0060] The ultrasonic microphone is used to receive the second frequency modulated continuous wave to obtain the lip ultrasonic echo signal.

[0061] A third aspect of an embodiment of the present invention discloses another lip feature fusion speech recognition device, the device comprising:

[0062] A memory storing executable program code;

[0063] a processor coupled to the memory;

[0064] The processor calls the executable program code stored in the memory to execute part or all of the steps in the lip feature fusion speech recognition method disclosed in the first aspect of the embodiment of the present invention.

[0065] The fourth aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the lip feature fusion speech recognition method disclosed in the first aspect of the embodiment of the present invention.

[0066] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0067] The ultrasonic speaker and the ultrasonic microphone are used to obtain the lip ultrasonic unimodal data features; the camera is used to obtain the lip image unimodal data features; the lip two-source modal feature fusion model is used to fuse the lip ultrasonic unimodal data features and the lip image unimodal data features to obtain multimodal speech information; based on the multimodal speech information, the speech recognition text and speech recognition synthesized audio are determined. It can be seen that the present application can overcome the recognition defects of single modal features, improve the naturalness, accuracy and robustness of the recognition results in silent mode or high noise intervention, and the algorithm complexity is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 This is a flow chart of a lip feature fusion speech recognition method proposed in the present invention.

[0070] Figure 2 A schematic diagram of a differential echo contour curve of a lip feature fusion speech recognition method proposed in the present invention.

[0071] Figure 3 This is a schematic diagram of the structure of a lip feature fusion speech recognition device proposed by the present invention.

[0072] Figure 4 This is a schematic diagram of the structure of a head-mounted support module and a data acquisition module of a lip feature fusion speech recognition device proposed in the present invention;

[0073] In the figure: 1. Camera; 2. Ultrasonic speaker; 3. Ultrasonic microphone; 4. Head-mounted bracket module.

[0074] Figure 5 This is a schematic diagram of the structure of an interactive module of a lip feature fusion speech recognition device proposed in the present invention.

[0075] Figure 6 This is a schematic diagram of the structure of an interaction module of another lip feature fusion speech recognition device proposed by the present invention. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0078] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0079] The present invention discloses a lip feature fusion speech recognition method and device, the method comprising: using an ultrasonic speaker and an ultrasonic microphone to obtain lip ultrasonic unimodal data features; using a camera to obtain lip image unimodal data features; using a lip two-source modal feature fusion model to fuse lip ultrasonic unimodal data features and lip image unimodal data features to obtain multimodal speech information; based on the multimodal speech information, determining speech recognition text and speech recognition synthesized audio. It can be seen that the present application solves the recognition defects of single modal features, improves the naturalness, accuracy and robustness of the recognition results in silent mode or high noise intervention, and the algorithm complexity is low. The following is a detailed description.

[0080] Embodiment 1

[0081] See also Figure 1 , Figure 1 : is a flow chart of a lip feature fusion speech recognition method disclosed in an embodiment of the present invention. Figure 1 The lip feature fusion speech recognition method described is applied to the field of speech interaction, such as communication between patients with voice disorders, communication in high-noise or hidden private scenes, etc., and the embodiments of the present invention do not limit this. Figure 1 As shown, the lip feature fusion speech recognition method may include the following operations:

[0082] S1, using a preset ultrasonic speaker and ultrasonic microphone to obtain lip ultrasonic single-modal data features;

[0083] S2, using a preset camera to obtain lip image single-modal data features;

[0084] Preferably, the camera is an RGB camera;

[0085] S3, using the lip two-source modal feature fusion model, the lip ultrasonic single modal data features and the lip image single modal data features are fused to obtain multimodal speech information;

[0086] S4, based on the multimodal speech information, determines the speech recognition text and the speech recognition synthesized audio.

[0087] In an optional embodiment, the above-mentioned use of a preset ultrasonic speaker and an ultrasonic microphone to obtain the lip ultrasonic single-mode data features includes:

[0088] S11, using an ultrasonic speaker to continuously emit a first frequency modulated continuous wave toward the lip area of ​​the user;

[0089] Preferably, the lowest frequency and the highest frequency of the first frequency modulated continuous wave are 20kHz and 25kHz respectively;

[0090] Preferably, the scanning period of the first frequency modulated continuous wave is 12ms;

[0091] It should be noted that the frequency of the first frequency modulated continuous wave changes linearly in each scanning period, and its frequency-time function is a triangular wave;

[0092] S12, using an ultrasonic microphone to receive a second frequency modulated continuous wave to obtain a lip ultrasonic echo signal;

[0093] It should be noted that the above-mentioned second frequency modulated continuous wave is a signal generated after the first frequency modulated continuous wave is reflected and diffracted by the lip area; after the second frequency modulated continuous wave is received by the ultrasonic microphone, the lip ultrasonic echo signal obtained is a digital signal, which is used to describe the change of signal strength with the sampling time.

[0094] S13, processing the first frequency modulated continuous wave and the lip ultrasonic echo signal to obtain a differential echo contour curve;

[0095] S14, using the first feature extraction model to process the differential echo contour curve to obtain the lip ultrasonic single-mode data features.

[0096] Optionally, the first feature extraction model is a convolutional neural network, including a 2D CNN network and a first Resnet-18 network; the input end of the 2D CNN network receives a differential echo contour curve; the output end of the 2D CNN network is connected to the first Resnet-18 network; the output end of the first Resnet-18 network outputs lip ultrasonic single-modal data features.

[0097] It should be noted that the above 2D CNN network includes a classic 2D convolution layer, a batch normalization layer, and a RELU activation function layer; the above first Resnet-18 network is a classic residual network structure, consisting of 4 groups, each with two residual blocks; the above residual block contains a 3×3 convolution layer;

[0098] Preferably, the total depth of the pooling layer, activation layer and linear layer of the above-mentioned first feature extraction model is 18.

[0099] In another optional embodiment, the first frequency modulated continuous wave and the lip ultrasonic echo signal are processed to obtain a differential echo contour curve, including:

[0100] S131, initializing the current window number to 0;

[0101] S132, determining a distance deviation signal based on the first frequency modulated continuous wave and the lip ultrasonic echo signal;

[0102] It should be noted that the above distance deviation signal includes the distance deviations at N1 sampling moments, which is used to describe the change of the distance deviation caused by lip movement with the sampling moment; the above N1 is an integer greater than 1;

[0103] S133, combining the distance deviation signal and the lip ultrasonic echo signal into an echo profile frame;

[0104] It should be noted that the above distance deviation signal and lip ultrasonic echo signal are both one-dimensional signals, and the echo profile frame is a two-dimensional signal; the distance deviation signal and the lip ultrasonic echo signal at the same sampling time are combined to obtain an echo profile frame, which is used to describe the distribution of the intensity of the frequency modulated continuous wave after it is reflected and diffracted by the intermediate medium (skin, muscle, etc. in the lip area) as the sampling time and distance deviation change; the signal length of the echo profile frame is N1;

[0105] S134, adding 1 to the current window number;

[0106] S135, repeating S132 to S134 until the current window number reaches N2, and obtaining N2 echo contour frames; N2 is a preset window number threshold; the above N2 echo contour frames constitute an echo contour frame sequence;

[0107] S136, sequentially subtracting adjacent echo contour frames in the echo contour frame sequence to obtain a differential echo contour frame sequence; the differential echo contour frame sequence includes differential echo contour frames;

[0108] S137, plotting all the differential echo contour frames in the differential echo contour frame sequence into the same two-dimensional coordinate system to obtain a differential echo contour curve.

[0109] It should be noted that if Figure 2 As shown, the horizontal axis of the differential echo profile curve is the sampling time, the vertical axis is the distance deviation, and the color depth represents the intensity.

[0110] In yet another optional embodiment, determining a distance deviation signal based on the first frequency modulated continuous wave and the lip ultrasonic echo signal comprises:

[0111] S1321, set the current number of cycles to 0;

[0112] S1322, using a lip ultrasonic wave similarity calculation model, processing the first frequency modulated continuous wave and the lip ultrasonic wave echo signal to obtain the lip ultrasonic wave similarity;

[0113] The above lip ultrasound similarity calculation model is:

[0114]

[0115] In the formula, v tx (m) and v tx (m+n) are the intensities of the first frequency modulated continuous wave at sampling time m and m+n, respectively, v rx (m) and v rx (m+n) are the intensities of the lip ultrasonic echo signals at sampling time m and m+n, R(n) is the value of the ultrasonic similarity at sampling time n, and N1 is the number of samples;

[0116] S1323, calculating the sampling time when the ultrasonic similarity reaches the peak value, and obtaining the delay frame number;

[0117] S1324, using a distance deviation calculation model, processing the delayed frame number to obtain a distance deviation;

[0118] The above distance deviation calculation model is:

[0119]

[0120] Where R is the distance deviation, C is the preset sound speed, Lag is the distance deviation, and F S is the preset sampling frequency;

[0121] S1325, adding 1 to the current number of cycles;

[0122] S1326, repeat S1322 to S1325 until the current number of cycles reaches N1, and N1 distance deviations are obtained.

[0123] In an optional embodiment, the step of sequentially subtracting adjacent echo contour frames in the echo contour frame sequence to obtain a differential echo contour frame sequence includes:

[0124] S1361, initializing the differential echo profile frame sequence to an empty sequence;

[0125] S1362, determining the current number of iterations p as the length of the echo contour frame sequence;

[0126] S1363, calculating the difference between the pth and p-1th echo contour frames in the echo contour frame sequence to obtain a current differential echo contour frame;

[0127] S1364, inserting the current differential echo profile frame into the differential echo profile frame sequence header;

[0128] S1365, reduce the current number of iterations p by 1;

[0129] S1366, repeat S1363 to S1365 until the current number of iterations is 0.

[0130] In yet another optional embodiment, the above-mentioned method of using a preset camera to obtain the lip image single-modal data features includes:

[0131] S21, using a camera to collect movements of the lip area to obtain a lip movement video;

[0132] S22, segmenting the lip movement video into a lip movement frame sequence;

[0133] S23, using the second feature extraction model to process the lip motion frame sequence to obtain lip image unimodal data features.

[0134] Optionally, the second feature extraction model is a convolutional neural network, including a 3D CNN network and a second Resnet-18 network; the input end of the 3D CNN network receives a lip motion frame sequence; the output end of the 3D CNN network is connected to the second Resnet-18 network; the output end of the second Resnet-18 network outputs unimodal data features of the lip image.

[0135] It should be noted that the above-mentioned 3D CNN network includes a classic 3D convolution layer, a batch normalization layer and a RELU activation function layer; the above-mentioned second Resnet-18 network adopts a classic residual network structure, which consists of 4 groups, each with two residual blocks.

[0136] In another optional embodiment, the above-mentioned lip two-source modality feature fusion model includes a first spatiotemporal feature extraction module, a second spatiotemporal feature extraction module, a spatiotemporal feature splicing module, a binary feature fusion module and a fusion feature processing module;

[0137] The first spatiotemporal feature extraction module is connected to the spatiotemporal feature splicing module, and is used to process the lip ultrasonic single-mode data features to obtain a first spatiotemporal feature vector, and send it to the spatiotemporal feature splicing module;

[0138] Optionally, the first spatiotemporal feature extraction module is a long short-term memory network (LSTM);

[0139] The second spatiotemporal feature extraction module is connected to the spatiotemporal feature splicing module, and is used to process the lip image unimodal data features to obtain a second spatiotemporal feature vector, and send it to the spatiotemporal feature splicing module;

[0140] Optionally, the second spatiotemporal feature extraction module is a long short-term memory network (LSTM);

[0141] The above-mentioned spatiotemporal feature concatenation module is connected to the binary feature fusion module, and is used to concatenate the first spatiotemporal feature vector and the second spatiotemporal feature vector to obtain an initial spatiotemporal feature vector, and send it to the binary feature fusion module;

[0142] The binary feature fusion module is connected to the fusion feature processing module to process the initial spatiotemporal feature vector to obtain the lip modality collaborative feature and send it to the fusion feature processing module;

[0143] Optionally, the binary feature fusion module is a bidirectional long short-term memory network (Bi-LSTM);

[0144] Preferably, in the above-mentioned bidirectional long short-term memory network, the state dimension and the number of layers of the hidden layer are 1024 and 2 respectively;

[0145] The above-mentioned fusion feature processing module is used to process the lip modal collaborative features to obtain multimodal speech information.

[0146] It should be noted that the above multimodal speech information is a vector composed of multiple probability values, which is used to represent the current text probability distribution;

[0147] Optionally, the fusion feature processing module is a classifier, including a linear layer and a softmax layer; the lip modality collaborative features are processed in sequence by the linear layer and the softmax layer to obtain multimodal speech information.

[0148] It should be noted that the LSTM is a special type of RNN, but its gate structure (input gate, forget gate, and output gate) solves the long-term dependency problem of the RNN, allowing the network itself to store long-term information. The forget gate of the LSTM determines the information that needs to be discarded, and its input is the output h of the previous layer. t-1 And the sequence data x input to this layer t . Its formula is described as follows:

[0149] f t =σ(W f ·[h t-1 , x t ]+b f )

[0150] Among them, σ represents the activation function sigmoid, b f is the offset, W f is the weight of the input of this layer. The input gate of the long short-term memory network consists of two parts. The first part uses sigmoid as the activation function and the output is i t , the second part uses tanh as the activation function, and the output is, and its formula is as follows:

[0151] i t =σ(W i ·[h t-1 , x t ]+b i )

[0152]

[0153] Among them, as the updated state of this layer, C t It can be expressed as:

[0154]

[0155] The output gate of the long short-term memory network is used to control the data state filtering of this layer. The output of this layer is:

[0156] h t ={σ(W o [h t-1 , x t ]+b o )}*tanh(C t )

[0157] Combining the forward LSTM and the backward LSTM gives us Bi-LSTM.

[0158] In yet another optional embodiment, the above-mentioned determining the speech recognition text and the speech recognition synthesized audio based on the multimodal speech information includes:

[0159] S41, processing the multimodal voice information to obtain a voice recognition text;

[0160] S42, converting the speech recognition text into initial speech audio;

[0161] Optionally, the speech recognition text is converted into initial speech audio through a vocoder;

[0162] S43, objectively evaluating the initial speech audio to obtain an objective evaluation result;

[0163] S44, performing a subjective evaluation on the initial speech audio to obtain a subjective evaluation result;

[0164] S45, judging the objective evaluation result and the subjective evaluation result, and obtaining the speech recognition synthesized audio;

[0165] When both the objective evaluation result and the subjective evaluation result are available, determining the speech recognition synthesized audio as the initial speech audio;

[0166] When the objective evaluation result or the subjective evaluation result is unavailable, it is determined that the speech recognition synthesized audio is empty.

[0167] It should be noted that the above-mentioned speech recognition synthesized audio is empty, indicating that no meaningful recognition results were obtained.

[0168] In yet another optional embodiment, the above-mentioned processing of the multimodal voice information to obtain the voice recognition text includes:

[0169] S411, obtaining an initial recognition text based on the multimodal voice information and a preset initial text library;

[0170] It should be noted that the above-mentioned initial text library includes more than one initial text; the above-mentioned initial text is a short sentence, an instruction or a logical character; each component in the multimodal speech information represents the probability value of the corresponding text in the initial text library;

[0171] Optionally, the above-mentioned obtaining of the initial recognition text based on the multimodal voice information and the preset initial text library is to select the largest component in the multimodal voice information and the corresponding initial text in the initial text library to obtain the initial recognition text;

[0172] S412, using the language model, processing the initial recognition text to obtain the speech recognition text.

[0173] Optionally, the above language model is an n-gram language model based on a beam search algorithm.

[0174] It should be noted that the above-mentioned initial recognition text is processed by the language model to obtain the speech recognition text, which helps to improve the naturalness, accuracy and robustness of the recognition results in silent mode or high noise intervention, and the algorithm complexity is low.

[0175] In yet another optional embodiment, the objective evaluation of the initial speech audio to obtain the objective evaluation result includes:

[0176] S431, obtaining an objective evaluation value based on the initial speech audio;

[0177] S432, judging the objective evaluation value;

[0178] When the objective evaluation value is greater than or equal to a preset first objective threshold value and less than or equal to a preset second objective threshold value, determining that the objective evaluation result is available;

[0179] When the objective evaluation value is less than the first objective threshold, or greater than the second objective threshold, the objective evaluation result is determined to be unusable.

[0180] In yet another optional embodiment, the objective evaluation value is obtained based on the initial speech audio to calculate the short-time objective intelligibility of the initial speech audio, and the objective evaluation value is a short-time objective intelligibility value.

[0181] It should be noted that Short-Time Objective Intelligibility is an objective speech intelligibility evaluation indicator. It predicts speech intelligibility by calculating the correlation between the time domain and frequency domain features of the speech signal. It is suitable for evaluating the effect of improving speech intelligibility in a noisy environment. The range of Short-Time Objective Intelligibility values ​​is from 0 to 1, and the higher the value, the higher the intelligibility.

[0182] In yet another optional embodiment, the above-mentioned subjective evaluation of the initial speech audio to obtain the subjective evaluation result includes:

[0183] S441, playing the initial voice audio;

[0184] S442, in response to the user's operation, obtaining a subjective evaluation result.

[0185] It should be noted that the initial voice audio is fed back to the user through playback, and the user confirms whether the information is accurate and the audio is clear after perception; when the information is accurate and the audio is clear, the user sets the subjective evaluation result as available, otherwise it is set to unavailable.

[0186] It can be seen that the implementation of the lip feature fusion speech recognition method described in the embodiment of the present invention is conducive to overcoming the recognition defects of single modal features, improving the naturalness, accuracy and robustness of the recognition results in silent mode or high noise intervention, and the algorithm complexity is low.

[0187] Embodiment 2

[0188] See also Figures 3 to 5 .in, Figure 3 The lip feature fusion speech recognition device described can be applied to the field of speech interaction, such as communication between patients with voice disorders, communication in high-noise or hidden private scenes, etc., and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the device may include a head-mounted support module, a data acquisition module, a data analysis module and an interaction module;

[0189] The head-mounted bracket module is worn on the user's head and is used to fix the data acquisition module above the lip area of ​​the user's face, and to fix the data analysis module and the interaction module on the side of the user's head;

[0190] Optionally, the head-mounted support module is a virtual reality glasses;

[0191] Optionally, the data acquisition module is hung below the user's eyes;

[0192] Preferably, the data analysis module and the interaction module are respectively fixed on both sides of the user's head;

[0193] The above-mentioned data acquisition module is used to collect lip ultrasonic echo signals and lip movement videos, and send them to the data analysis module;

[0194] The data analysis module is connected to the interaction module data, and is used to use the lip two-source modal feature fusion model to fuse the lip ultrasonic single-modal data features and the lip image single-modal data features to obtain multi-modal voice information, and send it to the interaction module;

[0195] The above-mentioned interaction module is used to determine the speech recognition text and the speech recognition synthesized audio based on the multimodal speech information.

[0196] In another optional embodiment, Figure 4 As shown, the above-mentioned data acquisition module includes a camera, an ultrasonic speaker and an ultrasonic microphone;

[0197] The camera is fixed just above the lip area through a head-mounted bracket module to collect lip movement video;

[0198] The ultrasonic speaker and ultrasonic microphone are fixed to both sides above the lip area through a head-mounted bracket module;

[0199] The ultrasonic speaker is used to emit a first frequency modulated continuous wave toward the lip area;

[0200] The ultrasonic microphone is used to receive the second frequency modulated continuous wave to obtain a lip ultrasonic echo signal.

[0201] In yet another optional embodiment, Figure 5 As shown, the above-mentioned interaction module includes a text generation unit, an initial speech generation unit, an objective evaluation unit, a subjective evaluation unit and an evaluation judgment unit;

[0202] The text generation unit is data-connected to the data analysis module and the initial speech generation unit, and is used to process the multimodal speech information, obtain speech recognition text, and send it to the initial speech generation unit;

[0203] The initial speech generation unit is connected to the objective evaluation unit, the subjective evaluation unit and the evaluation judgment unit, and is used to convert the speech recognition text into initial speech audio and send it to the objective evaluation unit, the subjective evaluation unit and the evaluation judgment unit;

[0204] Optionally, the initial speech generation unit is a vocoder;

[0205] The objective evaluation unit is connected to the evaluation and judgment unit and is used to objectively evaluate the initial speech audio to obtain an objective evaluation result;

[0206] The subjective evaluation unit is connected to the evaluation and judgment unit data, and is used to perform a subjective evaluation on the initial speech audio to obtain a subjective evaluation result;

[0207] Optionally, the subjective evaluation unit is provided with a microphone for playing initial voice audio to the user;

[0208] Optionally, the subjective evaluation unit is provided with a first button and a second button; when the first button is pressed first, the subjective evaluation result is available; when the second button is pressed first, the subjective evaluation result is unavailable;

[0209] The evaluation and judgment unit is used to judge the objective evaluation results and the subjective evaluation results, and obtain the speech recognition synthesized audio.

[0210] It can be seen that the implementation of the lip feature fusion speech recognition device described in the embodiment of the present invention is conducive to overcoming the recognition defects of single modal features, improving the naturalness, accuracy and robustness of the recognition results in silent mode or high noise intervention, and the algorithm complexity is low.

[0211] Embodiment 3

[0212] See also Figure 6 , Figure 6 is a schematic diagram of the structure of another lip feature fusion speech recognition device disclosed in an embodiment of the present invention. Figure 6 The lip feature fusion speech recognition device described can be applied to the field of speech interaction, such as communication between patients with voice disorders, communication in high-noise or hidden private scenes, etc., and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the lip feature fusion speech recognition device may include the following parts:

[0213] A memory 301 storing executable program codes;

[0214] a processor 302 coupled to the memory 301;

[0215] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the lip feature fusion speech recognition method described in the first embodiment.

[0216] Embodiment 4

[0217] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the lip feature fusion speech recognition method described in the first embodiment.

[0218] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0219] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, a magnetic disk storage, a magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0220] Finally, it should be noted that the lip feature fusion speech recognition method and device disclosed in the embodiment of the present invention only discloses the preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A lip feature fusion speech recognition method, characterized in that: include: S1, using a preset ultrasonic speaker and ultrasonic microphone to obtain lip ultrasonic single-modal data features; S2, using a preset camera to obtain lip image single-modal data features; S3, using a lip two-source modal feature fusion model to fuse the lip ultrasonic single modal data features and the lip image single modal data features to obtain multimodal speech information; S4, determining speech recognition text and speech recognition synthesized audio based on the multimodal speech information.

2. The lip feature fusion speech recognition method according to claim 1, characterized in that: The method of obtaining the lip ultrasonic single-mode data features by using a preset ultrasonic speaker and an ultrasonic microphone includes: S11, using an ultrasonic speaker to continuously emit a first frequency modulated continuous wave toward the lip area of ​​the user; S12, using an ultrasonic microphone to receive a second frequency modulated continuous wave to obtain the lip ultrasonic echo signal; S13, processing the first frequency modulated continuous wave and the lip ultrasonic echo signal to obtain a differential echo contour curve; S14, using a first feature extraction model to process the differential echo contour curve to obtain the lip ultrasonic single-mode data feature.

3. The lip feature fusion speech recognition method according to claim 2, characterized in that: The step of processing the first frequency modulated continuous wave and the lip ultrasonic echo signal to obtain a differential echo contour curve includes: S131, initializing the current window number to 0; S132, determining a distance deviation signal based on the first frequency modulated continuous wave and the lip ultrasonic echo signal; S133, combining the distance deviation signal and the lip ultrasonic echo signal into an echo contour frame; S134, adding 1 to the current window number; S135, repeating S132 to S134 until the current number of windows reaches N2, and obtaining N2 echo contour frames; N2 is a preset window number threshold; the N2 echo contour frames constitute an echo contour frame sequence; S136, sequentially subtracting adjacent echo contour frames in the echo contour frame sequence to obtain a differential echo contour frame sequence; the differential echo contour frame sequence includes differential echo contour frames; S137, plotting all the differential echo contour frames in the differential echo contour frame sequence into the same two-dimensional coordinate system to obtain a differential echo contour curve.

4. The lip feature fusion speech recognition method according to claim 1, characterized in that: The method of using a preset camera to obtain the lip image single-modal data features includes: S21, using the camera to collect the movement of the lip area to obtain the lip movement video; S22, dividing the lip movement video into a lip movement frame sequence; S23, using a second feature extraction model to process the lip motion frame sequence to obtain the lip image unimodal data features.

5. The lip feature fusion speech recognition method according to claim 1, characterized in that: The lip two-source modality feature fusion model includes a first spatiotemporal feature extraction module, a second spatiotemporal feature extraction module, a spatiotemporal feature splicing module, a binary feature fusion module and a fusion feature processing module; The first spatiotemporal feature extraction module is connected to the spatiotemporal feature splicing module, and is used to process the lip ultrasonic single-mode data features to obtain a first spatiotemporal feature vector, and send it to the spatiotemporal feature splicing module; The second spatiotemporal feature extraction module is connected to the spatiotemporal feature splicing module, and is used to process the lip image unimodal data features to obtain a second spatiotemporal feature vector, and send it to the spatiotemporal feature splicing module; The spatiotemporal feature concatenation module is connected to the binary feature fusion module and is used to concatenate the first spatiotemporal feature vector and the second spatiotemporal feature vector to obtain an initial spatiotemporal feature vector and send it to the binary feature fusion module; The binary feature fusion module is connected to the fusion feature processing module, and is used to process the initial spatiotemporal feature vector to obtain the lip modality collaborative feature, and send it to the fusion feature processing module; The fusion feature processing module is used to process the lip modal collaborative features to obtain multimodal speech information.

6. The lip feature fusion speech recognition method according to claim 1, characterized in that: The step of determining speech recognition text and speech recognition synthesized audio based on the multimodal speech information includes: S41, processing the multimodal voice information to obtain a voice recognition text; S42, converting the speech recognition text into initial speech audio; S43, objectively evaluating the initial speech audio to obtain an objective evaluation result; S44, performing a subjective evaluation on the initial speech audio to obtain a subjective evaluation result; S45, judging the objective evaluation result and the subjective evaluation result, and obtaining the speech recognition synthesized audio; When both the objective evaluation result and the subjective evaluation result are available, determining the speech recognition synthesized audio to be the initial speech audio; When the objective evaluation result or the subjective evaluation result is unavailable, it is determined that the speech recognition synthesized audio is empty.

7. The lip feature fusion speech recognition method according to claim 6, characterized in that: The objectively evaluating the initial speech audio to obtain an objective evaluation result includes: S431, obtaining an objective evaluation value based on the initial speech audio; S432, judging the objective evaluation value; When the objective evaluation value is greater than or equal to a preset first objective threshold value and less than or equal to a preset second objective threshold value, determining that the objective evaluation result is available; When the objective evaluation value is less than the first objective threshold, or greater than the second objective threshold, the objective evaluation result is determined to be unavailable.

8. A lip feature fusion speech recognition device, characterized in that: The device includes a head-mounted support module, a data acquisition module, a data analysis module and an interaction module; The head-mounted support module is worn on the user's head and is used to fix the data acquisition module above the lip area of ​​the user's face, and to fix the data analysis module and the interaction module on the side of the user's head; The data acquisition module is used to collect the lip ultrasonic echo signal and the lip movement video, and send them to the data analysis module; The data analysis module is data-connected to the interaction module, and is used to fuse the lip ultrasound single-modal data features and the lip image single-modal data features using a lip two-source modal feature fusion model to obtain multi-modal voice information, and send it to the interaction module; The interaction module is used to determine the speech recognition text and the speech recognition synthesized audio based on the multimodal speech information.

9. The lip feature fusion speech recognition device according to claim 8, characterized in that: The data acquisition module includes a camera, an ultrasonic speaker and an ultrasonic microphone; The camera is fixed directly above the lip area through the head-mounted bracket module to collect the lip movement video; The ultrasonic speaker and the ultrasonic microphone are respectively fixed on both sides above the lip area through the head-mounted bracket module; The ultrasonic speaker is used to transmit the first frequency modulated continuous wave to the lip area; The ultrasonic microphone is used to receive the second frequency modulated continuous wave to obtain the lip ultrasonic echo signal.

10. A lip feature fusion speech recognition device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the lip feature fusion speech recognition method as described in any one of claims 1-7.

Images