Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A continuous sign language recognition method

A sign language and image sequence technology, applied in character and pattern recognition, instruments, biological neural network models, etc., to achieve the effect of increasing feature dimension, increasing diversity, and predicting sequence accuracy

Active Publication Date: 2022-05-03
HEBEI UNIV OF TECH +1
View PDF11 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The technical problem to be solved by the present invention is to provide a method for continuous sign language recognition, which is a method for continuous sign language recognition based on multi-modal image sequence feature fusion and self-attention mechanism codec network, firstly obtain the optical flow image sequence , by extracting the spatio-temporal features of the original sign language image sequence and optical flow image sequence and fusion of the spatio-temporal features of the multi-modal image sequence, and extracting the text feature sequence of the sign sentence label, the fused multi-modal image sequence spatio-temporal features and the extracted sign language The text feature sequence of the sentence label is input into the encoding and decoding network based on the self-attention mechanism to predict the output of the sign language label, which overcomes the shortcomings of the existing technology that the feature is single and the video needs to be segmented

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A continuous sign language recognition method
  • A continuous sign language recognition method
  • A continuous sign language recognition method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0181] In the first step, the optical flow image sequence is obtained by extracting the original sign language image sequence through the FlowNet network:

[0182] Read in the video P01_s1_00_0_color.avi composed of n=228 shots, the video size is 112×112 pixels, and the input original sign language image sequence X=(x 1 ,x 2 ,...,x n ), wherein, n=228 is the frame number of the image sequence (the same below), x 1 、x 2 ,...,x n They are the first frame, the second frame, ..., the nth frame of the original sign language image sequence, and the optical flow field between adjacent images is extracted through the FlowNet network, and the optical flow field between each sign language image sequence forms an optical flow image sequence, The obtained optical flow image sequence containing n frames of images is X'=(x' 1 ,x' 2 ,...,x' n ), where x' 1 , x' 2 ,...,x' n Respectively, the first frame, the second frame, ..., the nth frame of the optical flow image sequence;

[01...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a continuous sign language recognition method, which relates to the processing of recording carriers used to recognize graphics, and is a method for continuous sign language recognition based on the encoding and decoding network of multi-modal image sequence feature fusion and self-attention mechanism. First, the optical flow is obtained Image sequence, through the extraction of the original sign language image sequence and optical flow image sequence spatio-temporal features and the fusion of multi-modal image sequence spatio-temporal features, and the extraction of the text feature sequence of the sign sentence label, the fusion of the multi-modal image sequence spatio-temporal features and extraction The text feature sequence of the sign language sentence label is input into the codec network based on the self-attention mechanism to predict the sign language label output, which overcomes the shortcomings of the existing technology that the feature is single and the video needs to be segmented.

Description

technical field [0001] The technical solution of the present invention relates to the processing of record carriers for recognizing graphics, in particular a continuous sign language recognition method. Background technique [0002] Hearing-impaired people have many inconveniences in daily life due to language barriers. Sign language recognition technology can help hearing-impaired people communicate with hearing people. The key technology of sign language recognition is to design a visual descriptor, which can reliably capture gestures, postures and facial expression features for sign language recognition. There are two research directions of sign language recognition technology at home and abroad, one is sensor-based data glove sign language recognition, and the other is sign language recognition based on visual features. Due to the inflexibility of sensor-based data glove sign language recognition equipment, it cannot be used in daily life. In recent years, the research...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06V10/80G06V40/20G06K9/62G06N3/04
CPCG06V40/28G06N3/045G06F18/253G06F18/214
Inventor 于明秦梦现薛翠红郝小可郭迎春阎刚于洋师硕刘依
Owner HEBEI UNIV OF TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com