Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Continuous sign language recognition method

A sign language, image sequence technology, used in character and pattern recognition, instrumentation, biological neural network models, etc.

Active Publication Date: 2020-06-26
HEBEI UNIV OF TECH +1
View PDF11 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The technical problem to be solved by the present invention is to provide a method for continuous sign language recognition, which is a method for continuous sign language recognition based on multi-modal image sequence feature fusion and self-attention mechanism codec network, firstly obtain the optical flow image sequence , by extracting the spatio-temporal features of the original sign language image sequence and optical flow image sequence and fusion of the spatio-temporal features of the multi-modal image sequence, and extracting the text feature sequence of the sign sentence label, the fused multi-modal image sequence spatio-temporal features and the extracted sign language The text feature sequence of the sentence label is input into the encoding and decoding network based on the self-attention mechanism to predict the output of the sign language label, which overcomes the shortcomings of the existing technology that the feature is single and the video needs to be segmented

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Continuous sign language recognition method
  • Continuous sign language recognition method
  • Continuous sign language recognition method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0181] In the first step, the optical flow image sequence is obtained by extracting the original sign language image sequence through the FlowNet network:

[0182] Read in the video P01_s1_00_0_color.avi composed of n=228 shots, the video size is 112×112 pixels, and the input original sign language image sequence X=(x 1 ,x 2 ,...,x n ), wherein, n=228 is the frame number of the image sequence (the same below), x 1 、x 2 ,...,x n They are the first frame, the second frame, ..., the nth frame of the original sign language image sequence, and the optical flow field between adjacent images is extracted through the FlowNet network, and the optical flow field between each sign language image sequence forms an optical flow image sequence, The obtained optical flow image sequence containing n frames of images is X'=(x' 1 ,x' 2 ,...,x' n ), where x' 1 , x' 2 ,...,x' n Respectively, the first frame, the second frame, ..., the nth frame of the optical flow image sequence;

[01...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a continuous sign language recognition method, and relates to processing of a recording carrier for identifying graphics. The method is a continuous sign language recognition method based on multi-modal image sequence feature fusion and a coding and decoding network of a self-attention mechanism. The method comprises the following steps: obtaining an optical flow image sequence; extracting spatial-temporal features of an original sign language image sequence and an optical flow image sequence and fusing the spatial-temporal features of a multi-modal image sequence; extracting a text feature sequence of a sign language sentence label; and inputting the fused multi-modal image sequence spatial-temporal features and the extracted text feature sequence of the sign language sentence label into the coding and decoding network based on a self-attention mechanism to perform sign language label prediction output, thereby overcoming the defects of single feature and videosegmentation requirement in the prior art.

Description

technical field [0001] The technical solution of the present invention relates to the processing of record carriers for recognizing graphics, in particular a continuous sign language recognition method. Background technique [0002] Hearing-impaired people have many inconveniences in daily life due to language barriers. Sign language recognition technology can help hearing-impaired people communicate with hearing people. The key technology of sign language recognition is to design a visual descriptor, which can reliably capture gestures, postures and facial expression features for sign language recognition. There are two research directions of sign language recognition technology at home and abroad, one is sensor-based data glove sign language recognition, and the other is sign language recognition based on visual features. Due to the inflexibility of sensor-based data glove sign language recognition equipment, it cannot be used in daily life. In recent years, the research...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06K9/00G06K9/62G06N3/04
CPCG06V40/28G06N3/045G06F18/253G06F18/214
Inventor 于明秦梦现薛翠红郝小可郭迎春阎刚于洋师硕刘依
Owner HEBEI UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com