Attention fusion-based online short video multi-modal emotion recognition method

An emotion recognition and short video technology, applied in the field of online short video multimodal emotion recognition based on attention fusion, can solve the problem of difficulty in learning the relationship between modalities, so as to improve the performance of emotion classification and improve the effect of emotion recognition. Effect

Active Publication Date: 2020-06-12
CHONGQING UNIV OF POSTS & TELECOMM
View PDF9 Cites 41 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although this method fully considers the differences in the characteristics of each mode, it is difficult to learn the interrelationships between modes.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Attention fusion-based online short video multi-modal emotion recognition method
  • Attention fusion-based online short video multi-modal emotion recognition method
  • Attention fusion-based online short video multi-modal emotion recognition method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. The described embodiments are only part of the implementation of the present invention. example, not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

[0035] A multi-modal emotion recognition method for online short videos based on attention fusion, such as figure 1 As shown, the method steps include:

[0036] S1: Obtain each single-mode feature in the short video, that is, text features, voice features, and image features;

[0037] S2: Use a bidirectional GRU network to preprocess each single-mode feature separat...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the field of natural language processing, deep learning and multi-modal sentiment analysis, in particular to an attention fusion-based online short video multi-modal sentimentrecognition method, which comprises the following steps of: obtaining each single-modal feature in a short video; preprocessing the features by adopting a bidirectional GRU to obtain modal internal information; obtaining each advanced modal feature in combination with the interaction between the modal internal information and the modals; determining the contribution degree of each mode accordingto an attention mechanism to obtain a total feature vector, and inputting the total feature vector into a softmax function to obtain a bidirectional GRU multi-mode emotion recognition model based on attention fusion; training the model, and inputting a to-be-recognized short video into the trained model to obtain an emotion recognition result; according to the multi-modal emotion recognition method and system, all single-modal features are well fused, emotion information expressed in the video is effectively mined, and therefore the accuracy and efficiency of multi-modal emotion recognition are improved.

Description

technical field [0001] The invention relates to the fields of natural language processing, deep learning, and multimodal emotion analysis, and in particular to an online short video multimodal emotion recognition method based on attention fusion. Background technique [0002] With the widespread popularization of the Internet, the scale of mobile Internet users continues to expand, and more and more people communicate through the Internet. Therefore, a large number of users' valuable comment information on people, events, products, etc. have been generated on the Internet. These comment information It expresses people's emotional color and emotional tendency. However, with the advancement of communication technology and the rapid rise of emerging social media (such as Douyin, Miaopai, Kuaishou, etc.), online short videos have attracted more and more attention from people, and people are used to expressing their emotions by taking short videos or opinion. With the increase ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06K9/62G06K9/00G06F16/35G06N3/04G10L15/26G10L25/63
CPCG06F16/35G10L15/26G10L25/63G06V20/41G06N3/045G06F18/241G06F18/253
Inventor 唐宏赖雪梅陈虹羽李珊珊
Owner CHONGQING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap