Unlock instant, AI-driven research and patent intelligence for your innovation.

A lip reading method based on adaptive semantic spatiotemporal graph convolutional network

A convolutional network and spatiotemporal graph technology, applied in the field of lip reading based on adaptive semantic spatiotemporal graph convolutional network, can solve the problem of low recognition accuracy and achieve high accuracy.

Active Publication Date: 2020-07-31
NAT UNIV OF DEFENSE TECH
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In view of this, the present invention provides a lip-reading method based on an adaptive semantic spatio-temporal graph convolutional network to solve the problem of low recognition accuracy in existing lip-reading recognition methods

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A lip reading method based on adaptive semantic spatiotemporal graph convolutional network
  • A lip reading method based on adaptive semantic spatiotemporal graph convolutional network
  • A lip reading method based on adaptive semantic spatiotemporal graph convolutional network

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments produced by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention. In addition, it should be noted that "the..." in the content of the specific embodiment only refers to the technical attributes or characteristics of the present invention.

[0045] In order to further improve the accuracy of lip reading recognition based on the existing technology, we added the extraction of local visual features that can represent the local subtle motion information of the lip and the semantic information of the lip contour. refer to figure 1...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention provides a lip reading method based on adaptive semantic spatio-temporal graph convolutional network, the method includes extracting local semantic visual features including lip contour, local micro motion and semantic information and extracting lip global motion information Global visual features, and the fused visual features after the fusion of the local semantic visual features and global visual features are used for lip reading recognition, because the fused visual features not only include the global motion information of the lips but also the local motion of the lips and subtle motion information and semantic information, so that the lip reading method provided by the present invention has higher accuracy.

Description

technical field [0001] The invention belongs to the technical field of computer vision and pattern recognition, and in particular relates to a lip reading method based on an adaptive semantic spatiotemporal graph convolutional network. Background technique [0002] Automatic Lip Reading (ALR), or Visual Speech Recognition (VSR), aims to decode spoken content from a video containing a speaker's lip movement. Due to its potential application value, it has received more and more attention in recent years. Machines with lip-reading capabilities could open up many new applications, such as making smartphone reception more accurate in noisy environments, assisting the hearing-impaired, and subtitling silent movies, among other applications. [0003] The lip-reading recognition method based on deep learning is currently a relatively good recognition method. In the current lip-reading recognition method based on deep learning, the convolutional neural network (CNN) model is mostly ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06K9/00G06K9/62G06N3/04G06N3/08
CPCG06N3/08G06V40/20G06N3/045G06F18/24
Inventor 刘丽陈小鼎盛常冲龙云利
Owner NAT UNIV OF DEFENSE TECH