Audio visual speech module based on residual network and bidirectional gating recurrent units

Abstract

The invention provides an audio visual speech module based on a residual network and bidirectional gating recurrent units. The audio visual speech module mainly includes a visual flow, an audio flow,a classifying layer and an audio-video fusion, wherein the audio-video fusion includes the following processes that in the visual flow or the audio flow, temporal dynamics are modelized by two layersof the bidirectional gating recurrent units (BRGU), and then BGRU outputs of two signal flows are connected in series and transferred to the classifying layer to fuse; the temporal dynamics are jointly modeled; and at last outputting from an Softmax layer is achieved, each frame is signed by the Softmax layer, and a signed sequence is based on a topmost average probability. The characteristics ofpixels and audio waveforms can be directly extracted at the same time, the audio visual speech module has the text recognition function in a large public context dataset, in the condition of high noise intensity, compared with a traditional audio visual speech module, accuracy of classifying is obviously improved.

Description

technical field [0001] The invention relates to the field of audio-visual speech recognition, in particular to an audio-visual speech model based on a residual network and a bidirectional gating cycle unit. Background technique [0002] With the substantial improvement of the performance of personal computers, human-computer interaction technology has gradually shifted from computer-centered to human-centered interaction methods. Under this background, audio-visual speech recognition technology has also developed rapidly. Audio-visual speech recognition technology is mainly used in telephone and communication systems. People can easily query and extract relevant information from remote database systems through voice commands; audio-visual speech recognition technology is also widely used in user interactive machines, voice notepads, etc. , business self-service processing platform and other equipment, greatly reducing labor costs; in terms of public security criminal investi...

AI Technical Summary

Problems solved by technology

[0004] Aiming at the problem that the recognition accuracy is not high under the condition of strong noise, the object of the present invention is to provide an audio-visual speech model based on the residual network and the bidirectional gated recurrent unit. The bidirectional gated recurrent unit (BGRU) is modeled, and then the BGRU outputs of the two signal streams are concatenated and sent to the classification layer for fusion, and then their temporal dynamics are jointly modeled, and finally output from a Softmax layer, Softmax Each frame is labeled by the layer, and the labeled sequence is based on the highest average probability

Images