Lip reading method based on multi-granularity spatiotemporal feature perception of event camera

Abstract

The invention discloses a lip reading method based on multi-granularity spatiotemporal feature perception of an event camera. The lip reading method comprises the following steps: 1, providing a lip reading technical scheme based on the event camera for the first time; 2, according to the characteristics of an event stream signal, converting original asynchronous signal stream data into a multi-time resolution event frame; 3, a double-flow network is constructed to extract spatial and temporal features of different granularities, fine temporal features are extracted from a high-temporal-resolution branch, and complete spatial features are extracted from a low-temporal-resolution branch; and 4, constructing a sequence model to carry out feature sequence decoding, and decoding the multi-granularity spatio-temporal features extracted by the feature extraction network into the probability of words corresponding to event stream signals. According to the lip reading scheme based on the event camera, the problems of low video time resolution, much visual redundant information, poor performance under an extreme illumination condition and high power consumption of equipment during actual deployment when a traditional camera is used for lip reading can be solved.

Description

technical field [0001] The invention belongs to the field of lip reading, in particular to a lip reading method based on event camera-based multi-granularity spatiotemporal feature perception. Background technique [0002] Lip-reading technology aims to decode the textual content of a speaker's speech from visual information about the movement of the speaker's lips. It has important applications in the fields of health care, auxiliary speech recognition in noisy environments, public security, and human-computer interaction. Lip-reading technology has attracted great attention from academia and industry in the past 40 years. The lip reading task is a very challenging task, which is embodied in the following five aspects: 1. The video based on traditional RGB has low temporal resolution and contains a lot of visual redundant information such as background; 2. Different speakers’ pronunciation habits and faces 3. Words with similar pronunciation are visually ambiguous; 4. Vide...

AI Technical Summary

Problems solved by technology

[0005] In order to solve the shortcomings of the above-mentioned prior art, the present invention proposes a lip-reading method based on multi-granularity spatio-temporal feature perception of event cameras, in order to perform lip recognition more accurately through event stream signals, thereby solving the problem of Lip reading based on traditional RGB cameras has the problems of low temporal resolution of video, a lot of visual redundant information, poor performance under extreme lighting conditions, and high power consumption of devices during actual deployment.

Images