Small sample deep learning multi-modal sign language recognition method based on key frame sampling
A technology of deep learning and recognition methods, applied in the field of human-computer interaction recognition, can solve the problems of small sample size and researchers' failure to capture time information, and achieve the effect of improving accuracy
Active Publication Date: 2020-09-15
NANJING UNIV OF POSTS & TELECOMM
View PDF2 Cites 3 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Therefore, in previous Chinese sign language recognition design work, due to the small sample size, few people considered the method of deep learning, so many researchers failed to capture the temporal information
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0026] see figure 1 , this embodiment provides a small sample deep learning multimodal sign language recognition method based on key frame sampling, which can successfully train the neural network on the isolated word sign language recognition data set under the condition of small samples, and finally on the DEVISIGN data set The present optimal effect is obtained, and it is convenient for engineering realization.
[0027] Specifically include the following steps,
[0028] Step S1, through the depth camera, such as Kinect, collect the human skeleton information, RGB data and corresponding depth data of the sign language personnel, use the optical flow algorithm to convert the RGB color video into a streamer video, and select the RGB color video, Depth depth video and light Streaming video as multimodal input;
[0029] Step S2. Perform skin color detection on each frame of the RGB color video to preserve the hands and face, then remove the influence of face pixels based on th...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
A small sample deep learning multi-modal sign language recognition method based on key frame sampling comprises the steps: multi-modal data input: selecting RGB color video data, depth data and optical flow data as multi-modal data sources; hand segmentation: extracting a face and a hand based on skin color detection of a plurality of color spaces RGB, YCrCb and HSV, and eliminating the influenceof face pixels according to depth data and human skeleton coordinates; providing and adopting a new key frame sampling method based on hand skeleton data and optical flow; data enhancement; residual neural network R(2 + 1)D training based on the data of the uniform sampling data; proposing and utilizing cross-modal model data to train a data R(2 + 1)D fine tuning network for key frame sampling; and multi-modal fusion. According to the small sample deep learning multi-modal sign language recognition method, a deep learning method is successfully used for the first time on a small-sample Chinesesign language database DEVISIGN-D; the sign language lexicon is accurately recognized; the accuracy of the small sample deep learning multi-modal sign language recognition method exceeds that of a traditional method; and the optimal effect at present is achieved.
Description
technical field [0001] The invention belongs to the technical field of human-computer interaction recognition, and in particular relates to a small-sample deep learning multi-modal sign language recognition method based on key frame sampling. Background technique [0002] Currently, the loss of hearing ability seriously affects the quality of life of the hearing impaired. It is difficult for the hearing-impaired to communicate with ordinary people, who have little knowledge of sign language. It is hoped that automatic sign language recognition can bridge the communication gap. [0003] Existing technologies capture high-dimensional data by deploying data gloves, color gloves, or depth cameras, and then extract relevant handcraft features, such as joint trajectories, facial expressions, and hand shape features, for the subsequent recognition process. In recent years, it has been demonstrated that features extracted with the help of deep neural networks are more expressive t...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06K9/00G06K9/62G06N3/04G06N3/08
CPCG06N3/08G06V40/28G06N3/047G06N3/045G06F18/241G06F18/2415G06F18/253
Inventor 王剑羽陈建新
Owner NANJING UNIV OF POSTS & TELECOMM