A microphone speech recognition system and method based on multi-modal audio-visual fusion

Abstract

The application discloses a microphone speech recognition system and method based on multi-modal audio-visual fusion, and belongs to the field of artificial intelligence and voice interaction technology; first, an audio module collects voice signals through a microphone, converts the voice into text by means of a cloud voice recognition API, and further maps words into 300-dimensional semantic vectors by using Word2Vec. A visual module extracts lip movement and log-Mel spectrum features, after the lip image is detected and normalized by Dlib, the image is sent into a 3D CNN and a dense space-time CNN to extract space-time features, supplemented by a spatial attention mechanism to highlight key areas, and finally the sequence visual features are extracted by a bidirectional GRU. At the same time, the audio signal is generated into a log-Mel spectrum graph, and the perception characteristics are enhanced through Mel filtering and logarithmic processing. The audio word vector, the lip movement feature and the log-Mel feature are spliced into a multi-modal fusion vector, which is sent into a CTC decoder, and the text is predicted by Beam Search decoding. In the training process, an Adam optimizer and a small batch training strategy are used to improve the model performance and generalization ability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and voice interaction, and particularly relates to a microphone voice recognition system and method based on multi-modal audio-visual fusion. BACKGROUND

[0002] With the rapid development of artificial intelligence and voice interaction technology, automatic speech recognition (ASR) systems have been widely used in smart home, car navigation, medical transcription, robot interaction and other fields. However, the performance of speech recognition in noisy environments has become a key bottleneck restricting its practical application. Traditional speech recognition technology mainly relies on a single audio modality, and its performance is significantly affected by factors such as background noise, reverberation, and speaker differences, resulting in a significant reduction in recognition accuracy in complex scenarios (such as streets, factories, and public places). The performance of the speech recognition system is reduced, and it can only be used in limited specific environments before being applied to real-world environments, resulting in low-quality services and a decrease in consumer expectations. SUMMARY

[0003] The present application is to solve the problem of significant performance decline of existing speech technology in complex noisy environments in the background art, and aims to provide a microphone voice recognition system and method based on multi-modal audio-visual fusion. Starting from the actual factors affecting speech recognition, a new multi-modal audio-visual fusion method with strong robustness is proposed. Through innovative multi-feature fusion and network optimization, an efficient solution is provided for noise-robust speech recognition, promoting the practicalization process of multi-modal technology in real scenarios. The technical solution of the present application not only can significantly improve the accuracy of speech recognition, but also can reliably run repeatedly in various application scenarios. Unlike traditional solutions, through the fusion of audio, vision and spectrum three modalities, combined with deep learning optimization architecture, a high-robustness, low-error-rate speech recognition system is realized.

[0004] To solve the technical problem, the technical solution of the present application is:

[0005] A microphone voice recognition method based on multi-modal audio-visual fusion, the method comprising:

[0006] Step S1: Real-time acquisition of voice signals by microphone and sending them to cloud voice recognition API for conversion to text word sequence; then, using a pre-trained Word2Vec model to map each word to a 300-dimensional word vector, if the word recognition fails, using a zero vector to fill, and multiple word vectors are compressed through average pooling or a fully connected layer to finally generate a one-dimensional audio semantic feature vector;

[0007] Step S2: First, the face and lip region are located using Dlib, and the lip is aligned with key points and normalized to generate a lip movement image sequence; the image sequence is input into a 3D CNN and a dense spatiotemporal CNN to extract lip movement visual features, and a spatial attention mechanism is used to focus on the key lip region, and finally a bidirectional GRU is used to extract temporal features; at the same time, the speech signal is converted into a log-Mel spectrum to enhance the perceptual characteristics of the audio, and a log-Mel spectrum feature is generated;

[0008] Step S3: The audio semantic feature vector, lip movement visual feature, and log-Mel spectrum feature are concatenated into a fusion feature vector, which is input into a CTC decoder for decoding, and the Beam Search algorithm is used to output the final text; during training, the Adam optimizer and the mini-batch training strategy are used to improve the accuracy and generalization ability of the model.

[0009] Further, the step S1 comprises:

[0010] S101: Input the audio signal, collect the speech signal in real time through the local microphone device, and transmit the speech data to the cloud speech recognition API for recognition processing;

[0011] S102: Generate a word list, the API converts the speech into a text word sequence, and outputs an identifiable word list;

[0012] S103: Word embedding vectorization, use the Google News corpus pre-trained Word2Vec model, similar context words in the vector space are close in distance, map each word in the word list output by the API to a 300-dimensional vector, if the API does not recognize a word, fill in a zero vector, if the word list contains multiple words, compress them into a one-dimensional feature vector through average pooling or a fully connected layer, which is the final output of the audio module.

[0013] Further, the step S2 comprises:

[0014] S201: Lip image preprocessing: use the Dlib linear classifier to locate the face and extract the face and lip region to generate a lip sequence image; align the lip region according to the key points, eliminate the influence of head posture changes, perform channel normalization on each frame of image, reduce the interference of light changes, standardize the image data, and perform horizontal flip on the sequence image during training to improve the generalization ability of the model;

[0015] S202: log-Mel spectrogram generation, 3-second speech signal is divided into frames with a 25ms window and a 10ms step, a total of 750 spectrograms are generated; each frame of spectrum is filtered through a Mel triangular filter to convert linear frequency to Mel scale and enhance the perceptual characteristics of speech; the Mel spectrum energy is logarithmized to obtain the log-Mel spectrogram, which enhances the feature distinction degree of high and low frequencies;

[0016] S203: Construct a visual feature extraction network, including:

[0017] 3D CNN module, 3D CNN adds three-dimensional convolution kernel parameters based on 2D CNN, so that the feature map in the continuous frame can be associated with the continuous frame of the previous layer and integrated into a single frame, finally realizing the extraction of motion information; process the lip movement sequence image, extract the space-time features through the 3D convolution kernel, combine batch normalization, ReLU activation and 3D maximum pooling layer, and output the initial feature map;

[0018]

[0019] wherein, is the weight parameter of the three-dimensional convolution kernel at position (p, q, r);

[0020] wherein, is the input feature of the previous layer;

[0021] Dense spatio-temporal CNN module: adopts a dense short connection structure, relieves gradient disappearance through a short path connection, reduces parameter quantity, and improves training efficiency, specifically including:

[0022] dense block: each block contains 6 layers of BN→ReLU→3D convolution structure, the interlayer feature map is transmitted through splicing to enhance feature reuse;

[0023] transition block: contains BN→ReLU→3D convolution→3D average pooling, which compresses the channel number and reduces the feature map size, and reduces the calculation amount;

[0024] spatial attention module: aggregate channel information through average pooling and maximum pooling layers to generate a spatial attention weight map, multiply the original feature map and the attention map element by element to highlight the key regions of the lips, and the calculation formula is as follows:

[0025] M S (F)=σ(f 7×7 ([AvgFool(F);MaxPool(F)])):

[0026] wherein σ is a Sigmoid function, f 7×7 is a 7×7 convolution operation;

[0027] Bidirectional GRU module: process the time sequence features, control the information flow through the update gate and the reset gate, capture the long time sequence dependence, and output the hidden state to form a unified visual time sequence feature vector as the final output of the visual module for subsequent multi-modal feature fusion.

[0028] Further, the step S3 comprises:

[0029] The audio semantic feature vector, the lip movement visual feature, and the log-Mel spectrum feature are spliced to form a multi-modal fusion feature matrix to provide comprehensive information for subsequent decoding.

[0030] A connection time sequence classification loss function is used to solve the problem of inconsistent input and output sequence length, maximize the probability of correct label sequence, and combine the beam search algorithm for decoding to generate the final text prediction result; during training, an Adam optimizer is used, the learning rate is set to 0.0001, and the small batch size is 8 to improve the stability of the model and prevent overfitting.

[0031] A microphone speech recognition system based on multi-modal audio-visual fusion, the system is applied to the method described in any of the above, the system comprises:

[0032] The audio feature extraction and word vector generation module: real-time collects the voice signal through the microphone and sends it to the cloud voice recognition API to convert it into a text word sequence; then, a pre-trained Word2Vec model is used to map each word to a 300-dimensional word vector, and if the word recognition fails, a zero vector is used to fill it; the word vectors of multiple words are compressed through average pooling or a fully connected layer, and finally a one-dimensional audio semantic feature vector is generated;

[0033] The visual feature extraction and log-Mel spectrum graph generation module: first, the Dlib is used to locate the face and lip region, and the key points of the lip are aligned and normalized to generate a lip movement image sequence; the image sequence is input into the 3D CNN and the dense space-time CNN to extract the lip movement visual feature, and the spatial attention mechanism is used to focus on the key area of the lip; finally, the bidirectional GRU is used to extract the time sequence feature; at the same time, the voice signal is converted into a log-Mel spectrum graph to enhance the perceptual characteristics of the audio, and the log-Mel spectrum feature is generated;

[0034] Multi-modal feature fusion and decoding module: the audio semantic feature vector, the lip movement visual feature, and the log-Mel spectrum feature are spliced into a fusion feature vector, which is input into the CTC decoder for decoding, and the Beam Search algorithm is used to output the final text; during the training process, the Adam optimizer and the small batch training strategy are used to improve the accuracy and generalization ability of the model.

[0035] A computer device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for microphone speech recognition based on multi-modal audio-visual fusion according to any one of the above when executing the program.

[0036] A computer readable storage medium, the computer readable storage medium has a computer program stored thereon, the program is executed by a processor to implement the method for microphone speech recognition based on multi-modal audio-visual fusion according to any one of the above.

[0037] Compared with the prior art, the advantages of the present application are that:

[0038] Improve the accuracy of speech recognition: through three modalities fusion (audio word vector, lip movement, log-Mel spectrogram) to realize complementation, make up for the deficiency of single mode, and significantly improve the accuracy of speech recognition.

[0039] Strong information complementarity: by introducing the log-Mel spectrogram as the third mode, the audio signal is converted into a visual spectrum, based on three modalities fusion (audio word vector, lip movement, log-Mel spectrogram) to realize complementation, enhance noise robustness, and use dense spatiotemporal 3D CNN to efficiently extract spatiotemporal features.

[0040] Improve the efficiency and real-time performance of the computer: use spatial attention mechanism and bidirectional GRU to optimize feature extraction and time series modeling, reduce redundant calculation, improve inference speed, and avoid overfitting.

[0041] Therefore, the purpose of the present application is to propose an innovative multi-modal audio-visual fusion method to improve the robustness and recognition accuracy of the system in view of the technical defects in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The technical roadmap of the method for microphone speech recognition based on multi-modal audio-visual fusion. DETAILED DESCRIPTION

[0043] The specific embodiments of the present application will be described below in conjunction with the embodiments:

[0044] It should be noted that the structures, proportions, sizes, etc. shown in the present specification are only used to cooperate with the content disclosed in the specification, so that people skilled in the art can understand and read, and are not used to limit the implementation conditions of the present application. Any modification of structure, change of proportion relationship or adjustment of size, which does not affect the effect and purpose that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application.

[0045] Meanwhile, the terms such as "upper", "lower", "left", "right", "middle", and "one" cited in the specification are only for the convenience of clear description, and are not intended to limit the scope of the application, and the change or adjustment of the relative relationship is also considered as the implementation of the application without substantial change of the technical content.

[0046] Embodiment 1:

[0047] The embodiment provides a microphone speech recognition method based on multi-modal audio-visual fusion, aiming at solving the problem of significant performance degradation of a traditional speech recognition system in a complex noise environment. Specifically, the performance of the traditional speech recognition system in the noise environment is unstable, the audio signal is seriously interfered by background noise, which leads to a significant reduction in recognition accuracy and limits the actual application range. The limitation of a single mode, the pure audio recognition depends on the acoustic signal and is easily affected by noise, and cannot maintain stability under low signal-to-noise ratio (SNR) conditions; pure vision is not affected by noise, but has insufficient ability to distinguish homophonic words, and is restricted by factors such as light and posture changes. The multi-modal fusion is insufficient, and the existing multi-modal speech recognition technology usually only combines audio and lip visual information, which leads to insufficient information complementarity in a noise environment and limited performance improvement.

[0048] Therefore, the embodiment provides a multi-modal audio-visual fusion microphone speech recognition system and method with strong robustness, which can effectively solve the performance degradation problem of the speech recognition system in a complex noise environment. The method is based on multi-modal fusion, combines a deep learning optimization architecture, solves the problems of noise interference, homophone ambiguity, and computational efficiency, significantly improves the recognition stability in high-noise scenes such as traffic and public places, and realizes a high-robustness, low-error-rate speech recognition system. For example Figure 1 The main technical scheme flow of the application is shown in the figure. As can be seen from the figure, from the perspective of multi-modal audio-visual fusion, the technical scheme mainly includes the following three steps:

[0049] Step 1: Audio feature extraction and word vector generation

[0050] (1) Input audio signal: real-time acquisition of speech signals through a local microphone device, and transmission of speech data to a cloud speech recognition API (application programming interface) for recognition processing.

[0051] (2) Generate a word list: the API converts speech into a text word sequence, and outputs an identifiable word list.

[0052] (3) Word embedding vectorization: using the Word2Vec model pre-trained on the Google News corpus, words with similar contexts are close in vector space, mapping each word in the word list output by the API to a 300-dimensional vector, filling in a zero vector if the API does not recognize a word, and if the word list contains multiple words, compressing them into a one-dimensional feature vector through average pooling or a fully connected layer as the final output of the audio module.

[0053] Step two: visual feature extraction (lip movement and log-Mel spectrogram)

[0054] (1) Lip image preprocessing: using Dlib linear classifier to locate face and extract face and lip region, generating lip sequence image (resolution 640x480, 30fps); aligning the lip region according to the key points, eliminating the influence of head posture changes, normalizing each frame of image, mean value is 0, variance is 1, reducing the interference of light change, standardizing the image data, training the sequence image horizontally, improving the generalization ability of the model.

[0055] (2) log-Mel (log-Mel) spectrogram generation: 3-second speech signal (sampling rate 16kHz, 48000 sampling points) is divided into frames with 25ms window and 10ms step, a total of 750 frames of spectrogram; each frame of spectrum is filtered through Mel triangular filter to convert linear frequency to Mel scale and enhance the perceptual characteristics of speech; taking the logarithm of the Mel spectrum energy to obtain the log-Mel spectrogram (size 40x750), enhancing the distinction between high and low frequency features.

[0056] (3) Visual feature extraction network:

[0057] 3D CNN (3D convolutional neural network) module: unlike 2D CNN, 3D CNN can effectively extract multi-dimensional features in lip-reading tasks (such as lip, tongue and teeth movement features), which achieves this function by encoding motion information in multiple consecutive images. Specifically, the formula is as follows: 3DCNN adds three-dimensional convolution kernel parameters to the 2D CNN (two-dimensional convolutional neural network), so that the feature map in the continuous frame can be associated with the continuous frame of the previous layer, and also be integrated into a single frame, finally realizing the extraction of motion information. Process lip sequence image, extract space-time features through 3D convolution kernel (size 7x7x3), combine batch normalization (BN), ReLU (linear rectifier function) activation and 3D max pooling layer, output initial feature map.

[0058]

[0059] wherein, is the weight parameter of the three-dimensional convolution kernel at position (p, q, r).

[0060] wherein, is the input feature of the previous layer.

[0061] Dense Spatio-Temporal CNN Module: Dense short connection structure is adopted to relieve gradient vanishing, reduce parameter quantity and improve training efficiency. Specifically, it includes:

[0062] Dense Block: Each block contains 6 layers of BN→ReLU→3D convolution structure (convolution kernel 3×3×3), and the feature maps between layers are transmitted by splicing to enhance feature reuse.

[0063] Transition Block: Contains BN→ReLU→3D convolution→3D average pooling, which compresses the channel number and reduces the feature map size, and reduces the calculation amount.

[0064] Spatial Attention Module: The channel information is aggregated by average pooling and max pooling layers to generate a spatial attention weight map. The original feature map and the attention map are multiplied element by element to highlight the key regions of the lips. The calculation formula is as follows:

[0065] M S (F)=σ(f 7×7 ([AvgFool(F);MaxPool(F)])):

[0066] wherein σ is the Sigmoid function, f 7×7 is a 7×7 convolution operation.

[0067] Bidirectional GRU (Gated Recurrent Unit) Module: Process time sequence features, control information flow through update gate and reset gate, capture long time sequence dependence, concatenate the hidden states of forward and backward GRU, and output as a visual feature vector.

[0068] Step three: multi-modal feature fusion and decoding

[0069] (1) Feature Splicing: The audio word vector (300 dimensions), lip movement feature vector (from dense spatio-temporal CNN) and log-Mel spectrum feature vector are spliced into a fusion matrix.

[0070] (2) CTC decoding and training:

[0071] The connection time classification (CTC) loss function is used to solve the problem of inconsistent input and output sequence length, maximize the probability of correct label sequence, and combine with beam search algorithm decoding to generate the final text prediction result. During training, the Adam (adaptive matrix estimation) optimizer (learning rate 0.0001) is used, and the small batch size is 8 to prevent overfitting.

[0072] It can be understood that:

[0073] Multi-modal audio-visual fusion method: the present application proposes an innovative multi-modal audio-visual fusion method, which fuses audio + vision + spectrum three modalities, wherein the spectrum is to convert the speech signal into log-Mel frequency domain feature, enhance the representation of high frequency and low frequency details, make up for the lack of frequency domain information of pure visual modal, aim to solve the limitations of single modal, noise interference, homonymy ambiguity and other problems, realize the high robustness and low error rate of speech recognition system, so as to form a solution of speech recognition system adapting to complex noise environment.

[0074] Efficient network architecture design: the present application adopts dense space-time CNN and spatial attention mechanism, greatly reduces parameter redundancy through short connection and bottleneck layer, and reduces calculation complexity; in addition, the bidirectional gate recurrent unit (Bi-GRU) is combined to process time sequence data, the connection time sequence classification (CTC) loss is used to solve the input-output alignment problem, and the coding accuracy is improved.

[0075] The three-modal fusion method is the core innovation point of the present application, the log-Mel spectrum graph is introduced as the third modal, the audio signal is converted into visual spectrum, the information complementarity in complex noise environment is improved, and the accuracy of the speech recognition system is enhanced. This key point has important technical value and application prospect, and is the most key and unique innovation content of the present application patent.

[0076] In an alternative embodiment, multi-modal Transformer fusion: one alternative is to use Transformer to encode audio (Log-Mel spectrum), lip movement video (spatial-temporal feature) and text (ASR output) respectively, to dynamically align the time sequence relationship of audio and visual features through cross attention mechanism, for example, to enhance the weight of lip movement feature in noise interference, to use large-scale unlabeled data for multi-modal pre-training, and to improve the generalization of feature representation. However, the calculation complexity is high, and the model needs to be optimized to be lightweight to adapt to edge devices.

[0077] In an alternative embodiment, dynamic gated multi-modal fusion: another alternative is to design a dynamic weight gate for each modal, to adaptively adjust the contribution proportion of audio, vision and sound spectrum according to the input signal-to-noise ratio (SNR), to introduce a lightweight noise classifier to detect the type of environmental noise (such as traffic, human voice) in real time, and to dynamically switch the fusion strategy. However, an efficient noise classifier needs to be designed to ensure real-time performance.

[0078] Embodiment 2:

[0079] The present application provides a microphone speech recognition system based on multi-modal audio-visual fusion, which can be used to implement the above-mentioned microphone speech recognition method based on multi-modal audio-visual fusion, specifically comprising:

[0080] Audio feature extraction and word vector generation module: real-time collection of voice signals through microphone and sending to cloud voice recognition API for conversion into text word sequence; then, pre-trained Word2Vec model is used to map each word into 300-dimensional word vector, zero vector is used for filling if word recognition fails, and multiple word vectors are compressed through average pooling or fully connected layer to generate one-dimensional audio semantic feature vector;

[0081] Visual feature extraction and log-Mel spectrum generation module: first, Dlib is used to locate face and lip region, and key point alignment and normalization processing is performed on the lip to generate lip movement image sequence; image sequence is input into 3D CNN and dense space-time CNN to extract lip movement visual features, and spatial attention mechanism is used to focus on key lip regions, and finally bidirectional GRU is used to extract time sequence features; at the same time, voice signal is converted into log-Mel spectrum to enhance the perceptual characteristics of audio, and log-Mel spectrum feature is generated;

[0082] Multi-modal feature fusion and decoding module: audio semantic feature vector, lip movement visual feature and log-Mel spectrum feature are spliced into fusion feature vector, which is input into CTC decoder for decoding, and Beam Search algorithm is used to output final text; Adam optimizer and small batch training strategy are used in training process to improve the accuracy and generalization ability of the model.

[0083] Embodiment 3

[0084] The embodiment provides a terminal device, which comprises a processor and a memory, the memory is used for storing a computer program, the computer program comprises program instructions, and the processor is used for executing the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), ready-to-program gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to realize corresponding method processes or corresponding functions. The processor in the embodiment of the application can be used for the operation of a microphone speech recognition method based on multi-modal audio-visual fusion, comprising the following steps:

[0085] Step S1: Real-time acquisition of voice signals by microphone and sending to cloud voice recognition API for conversion to text word sequence; then, use pre-trained Word2Vec model to map each word to a 300-dimensional word vector, if word recognition fails, use zero vector to fill, multiple word vectors are compressed by average pooling or fully connected layer, and finally generate a one-dimensional audio semantic feature vector;

[0086] Step S2: First, use Dlib to locate the face and lip region, and perform key point alignment and normalization processing on the lip, to generate a lip movement image sequence; input the image sequence into 3D CNN and dense spatio-temporal CNN to extract lip movement visual features, and focus on the key lip area through spatial attention mechanism, and finally extract time sequence features through bidirectional GRU; at the same time, convert the voice signal into a log-Mel spectrum graph to enhance the perceptual characteristics of the audio, and generate a log-Mel spectrum feature;

[0087] Step S3: Concatenate the audio semantic feature vector, lip movement visual feature and log-Mel spectrum feature into a fusion feature vector, and input it into the CTC decoder for decoding, and output the final text combined with the Beam Search algorithm; during the training process, use the Adam optimizer and small batch training strategy to improve the accuracy and generalization ability of the model.

[0088] Embodiment 4:

[0089] The embodiment provides a storage medium, specifically a computer readable storage medium (Memory), which is a memory device in a terminal device, used for storing programs and data. It can be understood that the computer readable storage medium herein can include an internal storage medium in the terminal device, and of course can also include an expansion storage medium supported by the terminal device. The computer readable storage medium provides a storage space, which stores an operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer readable storage medium herein can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory.

[0090] The one or more instructions stored in the computer readable storage medium can be loaded and executed by the processor to realize the corresponding steps of the method for microphone speech recognition based on multi-modal audio-visual fusion in the above embodiments; the one or more instructions in the computer readable storage medium are loaded and executed by the processor as follows:

[0091] Step S1: Real-time collection of voice signals through a microphone and sending to a cloud voice recognition API for conversion into a text word sequence; then, using a pre-trained Word2Vec model to map each word into a 300-dimensional word vector, if word recognition fails, using a zero vector for padding, and multiple word vectors are compressed through average pooling or a fully connected layer to finally generate a one-dimensional audio semantic feature vector;

[0092] Step S2: First, use Dlib to locate the face and lip region, and perform key point alignment and normalization processing on the lips to generate a lip movement image sequence; input the image sequence into a 3D CNN and a dense spatiotemporal CNN to extract lip movement visual features, and focus on the key lip region through a spatial attention mechanism, and finally extract the time sequence features through a bidirectional GRU; at the same time, convert the voice signal into a log-Mel spectrum graph to enhance the perceptual characteristics of the audio, and generate a log-Mel spectrum feature;

[0093] Step S3: Concatenate the audio semantic feature vector, lip movement visual feature, and log-Mel spectrum feature into a fusion feature vector, and input it into a CTC decoder for decoding, and output the final text combined with the Beam Search algorithm; during the training process, use the Adam optimizer and the small batch training strategy to improve the accuracy and generalization ability of the model.

[0094] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied in the medium.

[0095] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for implementing each flow or multiple flows and / or blocks Figure 1 The means for implementing each flow or multiple flows and / or blocks

[0096] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0097] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 of the flow or flows and / or blocks Figure 1 of the block or blocks specified in the flow.

[0098] The above described embodiments of the present application are merely given as examples and are not intended to limit the present application to the specific forms disclosed. The present application is intended to cover all modifications, equivalents and alternatives falling within the scope of the present application. The scope of the present application is defined by the appended claims.

[0099] Many other changes and modifications can be made to the application without departing from the spirit and scope of the application. It is intended to cover all such changes and modifications that are within the scope of the application, as claimed in the following claims.

Claims

1. A microphone speech recognition method based on multi-modal audio-visual fusion, characterized in that, The method comprises: Step S1: Real-time collection of voice signals by a microphone and sending to a cloud voice recognition API for conversion into a text word sequence; then, using a pre-trained Word2Vec model to map each word into a 300-dimensional word vector, and if word recognition fails, using a zero vector for padding, and multiple word vectors being compressed through average pooling or a fully connected layer to finally generate a one-dimensional audio semantic feature vector; Step S2: first, using Dlib to locate a face and a lip region, and performing key point alignment and normalization processing on the lip to generate a lip movement image sequence; inputting the image sequence into a 3D CNN and a dense spatiotemporal CNN to extract lip movement visual features, and focusing on a key lip region through a spatial attention mechanism, and finally extracting a time sequence feature through a bidirectional GRU; simultaneously, converting the voice signal into a log-Mel spectrum graph to enhance the perceptual characteristics of the audio, and generating a log-Mel spectrum feature; the step S2 comprises: S201: lip movement image preprocessing: using a Dlib linear classifier to locate a face and extract a face and a lip region to generate a lip sequence image; aligning the lip region according to key points, eliminating the influence of head posture changes, performing channel normalization on each frame of image, reducing light change interference, standardizing image data, and horizontally flipping the sequence image during training to improve the generalization ability of the model; S202: log-Mel spectrum graph generation, 3-second voice signals are divided into frames with a 25ms window and a 10ms step, and a total of 750 spectrum graphs are generated; each frame of spectrum is filtered through a Mel triangular filter to convert linear frequency into Mel scale and enhance the perceptual characteristics of the voice; the Mel spectrum energy is taken as a logarithm to obtain a log-Mel spectrum graph, and the discrimination degree of high-frequency and low-frequency features is enhanced; S203: constructing a visual feature extraction network, comprising: a 3D CNN module, 3D CNN adds three-dimensional convolution kernel parameters based on 2D CNN, so that the feature map in the continuous frame can be associated with the continuous frame of the previous layer and integrated into a single frame, and finally the motion information is extracted; processing the lip movement sequence image, extracting the space-time features through the 3D convolution kernel, combining batch normalization, ReLU activation and 3D maximum pooling layer, and outputting the initial feature map; wherein, is a weight parameter of the three-dimensional convolution kernel at position (p, q, r); wherein, is the input feature of the previous layer; a dense spatiotemporal CNN module: using a dense short connection structure, relieving gradient disappearance through a short path connection, reducing the parameter amount, and improving the training efficiency, specifically comprising: a dense block: each block contains 6 layers of BN→ReLU→3D convolution structure, the interlayer feature map is transmitted through splicing to enhance feature reuse; a transition block: containing BN→ReLU→3D convolution→3D average pooling, compressing the channel number and reducing the feature map size to reduce the calculation amount; a spatial attention module: aggregating channel information through average pooling and maximum pooling layers to generate a spatial attention weight graph, multiplying the original feature map and the attention graph element by element to highlight the key lip region, and the calculation formula is as follows: M S (F) = σ(f 7×7 ([AvgFool(F); MaxPool(F)])): where σ is a Sigmoid function, f 7×7 is a 7x7 convolution operation; The bidirectional GRU module processes the time sequence feature, controls the information flow through the update gate and the reset gate, captures the long time sequence dependency, and outputs the hidden state which is spliced to form a unified visual time sequence feature vector as the final output of the visual module for subsequent multi-modal feature fusion. Step S3: The audio semantic feature vector, the lip movement visual feature, and the log-Mel spectrum feature are spliced into a fusion feature vector, which is sent to a CTC decoder for decoding, and the Beam Search algorithm is combined to output the final text; during the training process, an Adam optimizer and a small batch training strategy are used to improve the accuracy and generalization ability of the model.

2. The microphone speech recognition method based on multi-modal audio-visual fusion according to claim 1, characterized in that, The step S1 comprises: S101: Input the audio signal, collect the voice signal in real time through the local microphone device, and transmit the voice data to the cloud voice recognition API for recognition processing; S102: Generate a word list, the API converts the voice into a text word sequence, and outputs an identifiable word list; S103: Word embedding vectorization, use the Google News corpus pre-trained Word2Vec model, similar context words are close in vector space, map each word in the API output word list to a 300-dimensional vector, if the API does not recognize a word, fill in the zero vector, if the word list contains multiple words, compress them into a one-dimensional feature vector through average pooling or a fully connected layer, as the final output of the audio module.

3. The microphone speech recognition method based on multi-modal audio-visual fusion according to claim 1, characterized in that, The step S3 comprises: The audio semantic feature vector, the lip movement visual feature, and the log-Mel spectrum feature are spliced to form a multi-modal fusion feature matrix, providing comprehensive information for subsequent decoding; a connection time sequence classification loss function is used to solve the problem of inconsistent input and output sequence length, and maximize the probability of the correct label sequence; the Beam Search algorithm is combined for decoding to generate the final text prediction result; during training, the Adam optimizer is used, the learning rate is set to 0.0001, and the small batch size is 8, to improve the stability of the model and prevent overfitting.

4. A microphone speech recognition system based on multi-modal audio-visual fusion, characterized in that, The system is applied to the method of any one of claims 1-3, and the system comprises: An audio feature extraction and word vector generation module: real-time collection of voice signals through a microphone, and sending the voice signals to a cloud voice recognition API for conversion into a text word sequence; then, a pre-trained Word2Vec model is used to map each word to a 300-dimensional word vector, and if a word fails to be recognized, a zero vector is used for padding, and multiple word vectors are compressed through average pooling or a fully connected layer to finally generate a one-dimensional audio semantic feature vector; A visual feature extraction and log-Mel spectrum graph generation module: first, the Dlib is used to locate the face and lip region, and the lip region is subjected to key point alignment and normalization processing to generate a lip movement image sequence; the image sequence is input into a 3D CNN and a dense space-time CNN to extract the lip movement visual feature, and a spatial attention mechanism is used to focus on the key lip region, and finally a bidirectional GRU is used to extract the time sequence feature; meanwhile, the voice signal is converted into a log-Mel spectrum graph to enhance the perceptual characteristics of the audio, and the log-Mel spectrum feature is generated; A multi-modal feature fusion and decoding module: the audio semantic feature vector, the lip movement visual feature and the log-Mel spectrum feature are spliced into a fusion feature vector, which is sent into a CTC decoder for decoding, and the Beam Search algorithm is combined to output the final text; in the training process, an Adam optimizer and a small batch training strategy are adopted to improve the accuracy and generalization ability of the model.

5. A computer device, comprising: The computer readable storage medium stores a computer program, which is executed by the processor to implement the multi-modal audio-visual fusion based microphone speech recognition method in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which is executed by the processor to implement the multi-modal audio-visual fusion based microphone speech recognition method in any one of claims 1 to 3.

Images