Multi-order feature dynamic fusion sign language translation method based on data self-driving
A sign language translation and self-driven technology, which is applied to instruments, character and pattern recognition, computer components, etc., can solve the problems of irreversible negative effects, poor coherence and low accuracy of sign language translation
Active Publication Date: 2020-06-09
HEFEI UNIV OF TECH
View PDF6 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
The negative impact of bad features cannot be eliminated, resulting in poor coherence and low accuracy of sign language translation results
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0052] In this example, if figure 1 As shown, a data self-driven multi-level feature dynamic fusion sign language translation method includes: first extracting various visual and action features of the input sign language video, constructing a feature pool according to the multivariate correlation between features, and according to the model in each The recognition probability under the feature selects the optimal feature; then uses the data self-driven attention mechanism to optimize the feature to obtain the visual and action attention features; then performs the first-order dynamic feature fusion to obtain the fusion feature to mine visual features and action features Complementary information between them; then the second-order dynamic feature fusion is performed, and the score fusion weight of the hidden state under different features is dynamically calculated according to the task state, and finally the sign language translation sentence is obtained through score fusion; ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a multi-order feature dynamic fusion sign language translation method based on data self-driving, and the method comprises the steps: firstly extracting a plurality of types ofvisual and motion features of an inputted sign language video, constructing a feature pool according to the multivariate correlation between the features, and selecting an optimal feature according to the recognition probability of a model under each feature; performing feature optimization by using a data self-driven attention mechanism to obtain visual and action attention features; performingfirst-order dynamic feature fusion to obtain fusion features so as to mine complementary information between the visual features and the action features; then performing second-order dynamic feature fusion, dynamically calculating score fusion weights of hidden states under different features according to task states, and finally obtaining sign language translation sentences through score fusion.According to the invention, continuous sentence translation of the sign language video can be realized, and the accuracy of sign language recognition is improved.
Description
technical field [0001] The invention belongs to the field of multimedia information processing, and relates to technologies such as computer vision, natural language processing, and deep learning. Specifically, it is a multi-level feature dynamic fusion sign language translation method based on data self-driving. Background technique [0002] Early sign language recognition research focused on discrete sign language recognition, which is essentially a special video classification problem. With the development of video understanding technology, continuous sign language translation has attracted more and more attention. [0003] Due to the complexity of video data, existing sign language translation methods have many drawbacks, especially in the aspect of multi-feature fusion. Classic fusion methods are divided into front-end fusion and back-end fusion. Front-end fusion is performed at the feature level, while back-end fusion is performed at the decision level. Front-end fu...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06K9/00G09B21/00
CPCG09B21/00G06V40/28G06V20/41G06V20/46
Inventor 郭丹宋培培刘祥龙汪萌
Owner HEFEI UNIV OF TECH