Prisoner emotion recognition method for multi-modal feature fusion based on self-weight differential encoder

A technology of differential coding and feature fusion, applied in the field of emotional computing, can solve the problems of difficult to accurately judge the true emotions of prisoners, low recognition rate, poor robustness, etc., to eliminate degradation problems, improve expressive ability, and improve accuracy Effect

Active Publication Date: 2020-02-04
SHANDONG UNIV
View PDF6 Cites 20 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Due to the strong concealment of inmates' behavior clues and serious psychological precautions, relying on single-modal data for emotion recognition may generate a lot of noise, making it difficult to accurately judge the true emotions of inmates, and single-modal emotion recognition exists The characteristics of low rate and poor robustness

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Prisoner emotion recognition method for multi-modal feature fusion based on self-weight differential encoder
  • Prisoner emotion recognition method for multi-modal feature fusion based on self-weight differential encoder
  • Prisoner emotion recognition method for multi-modal feature fusion based on self-weight differential encoder

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0089] A method for emotion recognition of inmates based on self-weight differential encoder for multi-modal feature fusion, such as figure 2 shown, including the following steps:

[0090] (1) Data preprocessing: preprocess the data of the three modalities of text, voice and micro-expression, including text data, voice data, and micro-expression data, so that it meets the input requirements of the corresponding models of different modalities;

[0091] Text data refers to the text data of prison inmates’ conversations with family members / relatives and friends during remote video meetings; voice data refers to the audio data of prison inmates’ conversations with family members / relatives and friends during remote video meetings; micro-expression data refers to prison Facial micro-expression data of inmates during remote video meetings with family members / relatives and friends.

[0092] (2) Feature extraction: extract the emotional information contained in the data of the three ...

Embodiment 2

[0109] A method for emotion recognition of inmates based on self-weight differential encoder for multi-modal feature fusion according to Embodiment 1, the difference is: in the step (1),

[0110] For text data, the preprocessing process includes: segmenting the text data into words, and converting the text data into a data structure that the TextCNN model can receive and calculate according to the word segmentation results and the word vectors corresponding to the words.

[0111] In the process of data conversion, all text data including each word is numbered, and a dictionary is generated. The content in the dictionary is that each word corresponds to a serial number, and then each text is segmented, according to the serial number corresponding to the word in the dictionary. The text is converted into a mathematical sequence composed of a series of serial numbers, and then the serial number corresponds to the initialized word vector list, and the sequence is converted into mat...

Embodiment 3

[0126] According to the method for emotion recognition of inmates who carry out multi-modal feature fusion based on self-weight differential encoder according to embodiment 1, the difference is: in the step (2),

[0127] For text data, the feature extraction process includes: extracting the features of the text data through the TextCNN model;

[0128] The TextCNN model uses multiple kernels of different sizes to extract key information in sentences, so that it can better capture local correlations. The biggest advantage of TextCNN is its simple network structure. In the case of a simple model network structure, the introduction of trained word vectors has a very good effect, so that our model can accelerate the convergence speed while having a good effect.

[0129] For speech data, the feature extraction process includes:

[0130] c. Run OpenSMILE on the Linux operating platform, take the voice file in WAV format as input, select cmobase2010.conf as the standard feature data ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a prisoner emotion recognition method for multi-modal feature fusion based on a self-weight differential encoder, and the method comprises the following steps: (1) data preprocessing: carrying out preprocessing of text data, voice data and micro-expression data, and enabling the text data, the voice data and the micro-expression data to meet the input requirements of models corresponding to different modals; (2) feature extraction: respectively extracting emotion information contained in the preprocessed data of the three modes of text, voice and micro-expression to obtain corresponding feature vectors; (3) feature fusion: carrying out feature fusion on the feature vectors by adopting a self-weight differential encoder; and (4) training the model to obtain an optimal emotion recognition model.Multi-modal feature fusion is carried out by using the self-weight differential encoder, and through cross complementation of multiple modal features, the limitation of single-modal data and the negative influence of error information are effectively reduced, so that the extracted emotion features are richer, more effective and more accurate, and the emotion recognition effect of the prisoner is improved.

Description

technical field [0001] The invention relates to an emotion recognition method for inmates based on self-weight differential encoder for multimodal feature fusion, and belongs to the technical field of emotion calculation. Background technique [0002] Since the end of the 20th century, emotion has played an increasingly important role in the cognitive process. Contemporary cognitive scientists compare emotion with classic cognitive processes such as perception, learning, memory, and speech. Research on emotion itself and the interaction between emotion and other cognitive processes has become a research hotspot in contemporary cognitive science. Emotion recognition is also become an emerging field of research. [0003] The application of emotion recognition in daily life is to calculate the target person's emotion by computer when the target person's emotion is naturally revealed. It plays an irreplaceable role in many fields. For example, in information appliances and sma...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06K9/62G06K9/00G06N3/04
CPCG06V20/41G06V20/46G06N3/045G06F18/241
Inventor 李玉军张文真贲晛烨刘治朱孔凡
Owner SHANDONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap