Audio/video keyword identification method based on decision-making level fusion

Abstract

The invention relates to an audio/video keyword identification method based on decision-making level fusion. The method mainly includes the following steps that (1) a keyword audio/video is recorded, a keyword and non-keyword voice acoustic feature vector sequence and a visual feature vector sequence are obtained, and accordingly a keyword and non-keyword acoustic template and a visual template are trained; (2) acoustic likelihood and visual likelihood are obtained according to the audio/video in different acoustic noise environments, so that the acoustic mode reliability, visual mode reliability and optimal weight are obtained, and accordingly an artificial neural network can be trained; (3) secondary parallel keyword identification based on the acoustic mode and the visual mode is conducted on the audio/video to be detected according to the acoustic template, the visual template and the artificial neural network. According to the audio/video keyword identification method based on decision-making level fusion, the acoustic function and the visual function are fused at a decision-making level, the secondary parallel keyword identification based on the dual modes is conducted on the audio/video to be detected, the contribution of visual information in the acoustic noise environment is fully utilized, and therefore identification performance is improved.

Description

technical field [0001] The invention belongs to the field of information technology and relates to an audio and video processing technology applied in the field of human-computer interaction, in particular to an audio and video keyword recognition method based on decision-making layer fusion. Background technique [0002] As an important branch of continuous speech recognition, the purpose of keyword recognition technology is to detect the preset keywords in the continuous unrestricted speech stream. Since there is no need to decode the complete voice stream, keyword recognition is more flexible than continuous speech recognition, and is very suitable for some specific application fields, such as national defense monitoring, human-computer interaction, audio document retrieval, etc. In order to improve the robustness of speech recognition systems in noisy environments, in recent years, audio-video speech recognition technology has become a popular research direction by fusin...

AI Technical Summary

Problems solved by technology

[0006] To sum up, the current keyword recognition technology based on audio-video fusion mainly uses appearance features as visual features, while the existing appearance feature extraction methods mainly consider the characteristics of the mouth region from a global perspective, ignoring the description of time domain and Local information of local changes in the airspace, and these local information are crucial

In addition, the fusion strategy of audio and video adopts feature layer fusion. This method requires more training data to fully train a classifier, and it cannot solve the problem of visual and acoustic contribution distribution under different acoustic signal-to-noise ratio environments, which affects recognition. performance

Images