Emotion recognition method and system, electronic equipment, storage medium and program product

Abstract

The invention discloses an emotion recognition method and system, electronic equipment, a storage medium and a program product in the technical field of artificial intelligence, and the method comprises the steps: extracting text feature data and target type features from emotion reaction data of a target object perceived by a user, and generating personalized features according to personality test questionnaire data of the user. Aligning various time sequence features of the target type features in a time dimension, and fusing various time sequence features under various time steps based on an attention mechanism to obtain fused emotional reaction features; and carrying out dimension reduction processing on the fused emotion reaction features, the text feature data and the personalized features to obtain multi-modal feature data for emotion recognition. The emotion recognition method and device can solve the problem that the emotion recognition precision cannot meet user requirements due to insufficient consideration of personalized factors in related technologies, and in the emotion recognition process, the personalized factors are fully considered, so that the emotion recognition precision is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an emotion recognition method, system, electronic device, non-volatile storage medium and computer program product. Background Art

[0002] Emotion recognition is the process by which a computer analyzes and processes the user's visual and / or auditory data collected by sensors to determine the user's emotional state.

[0003] Individuals will show different emotional responses to the same stimulus based on their unique personality traits. In order to more accurately identify the user's emotional state, related technologies use the user's physiological data, such as EEG data, as the user's unique individual characteristics when performing emotion recognition. However, personalized emotion recognition based on physiological signals cannot accurately identify the user's emotional state.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present invention provides an emotion recognition method, system, electronic device, non-volatile storage medium and computer program product, which fully consider personalized factors in the emotion recognition process and effectively improve the emotion recognition accuracy.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: The present invention provides an emotion recognition method, comprising: Extract text feature data and target type features from the emotional response data of the user's perception of the target object; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; generate personalized features based on the user's personality test questionnaire data; align the various time series features of the target type features in the time dimension, and fuse the various time series features at each time step based on the attention mechanism to obtain a fused emotional response feature; perform dimensionality reduction processing on the fused emotional response feature, the text feature data, and the personalized feature to obtain multimodal feature data for emotion recognition.

[0007] In a first exemplary embodiment, the aligning of various types of time series features of the target type features in the time dimension includes: rearranging the feature dimensions of various types of time series features of the target type features to replace the second dimension and the third dimension of the feature vectors corresponding to various types of time series features; performing an adaptive maximum pooling operation on the new third dimension for each type of time series features after the dimension replacement, and outputting various types of time series features with the same third dimension; and rearranging the feature dimensions of each type of time series features again according to the dimensional output format of the target type features.

[0008] In a second exemplary implementation, before aligning various types of time series features of the target type features in the time dimension, it also includes: when a network scale adjustment instruction is received, parsing the network scale adjustment instruction to obtain a time step hyperparameter; based on the time step hyperparameter, performing an adaptive maximum pooling operation on the various types of time series features after the permutation of dimensions, to obtain various types of time series features whose third dimension is the time step hyperparameter.

[0009] In a third exemplary embodiment, the generating of personalized features based on the user's personality test questionnaire data includes: when a personalized data collection request is detected, sending and displaying a personality test questionnaire page on a user terminal pre-bound by the user; when it is detected that the data is filled in, generating a personality test result based on the personality test data on the personality test questionnaire page; when a test result sending instruction is received, encrypting and transmitting the personality test result, and converting the received personality test result into a personalized feature according to the personalized data input format.

[0010] In a fourth exemplary implementation, before the personality test questionnaire page is sent and displayed on the user terminal pre-bound by the user, it also includes: pre-building a personality test questionnaire table, and setting the display trigger condition of the personality test questionnaire page to a personalized data collection request; wherein the personality test questionnaire table includes a first personality option, a second personality option, a third personality option, a fourth personality option and a fifth personality option; the first personality option, the second personality option, the third personality option, the fourth personality option and the fifth personality option all have personality scores, a first personality aspect, a second personality aspect and a third personality aspect; each first personality aspect, each second personality aspect and each third personality aspect has a corresponding personality aspect score; obtaining personality test text data including Big Five personality test questions and scores of each question, and filling the personality test questionnaire table based on the personality test text data to generate a personality test questionnaire page.

[0011] In a fifth exemplary embodiment, the attention-based mechanism is used to fuse various types of time series features at each time step to obtain a fused emotional response feature, including: obtaining implicit state data of various types of time series features of the target type features at each time step; performing multi-head attention calculation on each current implicit state data of the current time step to obtain a multi-head weighted vector corresponding to each type of time series feature; based on the multi-head weighted vector corresponding to each type of time series feature, performing dimensionality reduction processing on each type of time series feature, and normalizing the concatenated vectors of the various types of time series features after dimensionality reduction to obtain mixed time series data of the current time step; and obtaining a fused emotional response feature based on the mixed time series features of each time step.

[0012] In a sixth exemplary embodiment, the multi-head attention calculation is performed on each current implicit state data of the current time step, including: splicing each current implicit state data into an implicit state splicing vector, and multi-head copying the implicit state splicing vector according to the number of heads; each head independently and in parallel calculates the attention score: multi-head weight pre-calculation of the implicit state splicing vector to obtain a weighted pre-vector; based on the feature dimension of each current implicit state data, the implicit state splicing vector is divided into a plurality of sub-implicit state splicing vectors corresponding to the number of heads; for each head based on the weighted pre-vector, the attention score of each sub-implicit state splicing vector is calculated respectively, and each attention score is spliced ​​to obtain a total attention score; the attention weighted outputs of each head are spliced ​​to obtain a weighted total vector, and the multiple weighted total vectors are divided in the order of each type of time series features to obtain multi-head weighted vectors corresponding to each type of time series features.

[0013] Another aspect of the present invention provides an emotion recognition method, comprising: Obtain an emotion recognition training sample set and build an emotion recognition model; the emotion recognition model includes an input layer, a multimodal feature processing submodel and an emotion recognition submodel, and the multimodal feature processing submodel includes a feature extraction layer, an adaptive alignment module, a temporal hybrid attention module and a feature dimensionality reduction module.

[0014] The emotion recognition model is cyclically trained using the emotion recognition training sample set based on a target classification loss function until a preset model training stop condition is reached.

[0015] The emotional response data of the user perceiving the target object and the personality test questionnaire data of the user are input into the trained emotion recognition model, and the input layer is used to receive the emotional response data of the user perceiving the target object and the personality test questionnaire data of the user; the feature extraction layer extracts text feature data and target type features from the emotional response data, and generates personalized features according to the personality test questionnaire data of the user; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; the adaptive alignment module aligns various types of time series features of the target type features in the time dimension; the time series mixed attention module fuses various types of time series features at each time step based on the attention mechanism to obtain a fused emotional response feature; the feature dimensionality reduction module performs dimensionality reduction processing on the fused emotional response feature, the text feature data and the personalized feature; the emotion recognition sub-model splices the emotional response compression feature, the text compression feature and the personality compression feature into multimodal feature data, performs emotion recognition on the multimodal feature data, and outputs the emotion recognition result.

[0016] In a first exemplary embodiment, the feature extraction layer includes one or more of an audio feature extraction model, a face recognition model, a visual feature extraction model, and a language feature extraction model; the extraction of text feature data and target type features from the emotional response data includes: inputting the emotional response video data into the audio feature extraction model, and obtaining audio timing features according to the output of the audio feature extraction model; inputting the emotional response video data into the face recognition model, performing face recognition and face feature extraction according to the face recognition model, and obtaining face timing features; inputting the emotional response video data into the visual feature extraction model, and obtaining video timing features according to the output of the visual feature extraction model; converting the audio data of the emotional response video data into text data, and inputting the text data into the language feature extraction model, and obtaining text feature data according to the output of the language feature extraction model.

[0017] In a second exemplary embodiment, the feature dimensionality reduction module includes a text information compression model; the text information compression model includes a first fully connected layer and a second fully connected layer; the number of neurons in the first fully connected layer matches the feature dimension of the text feature data, and the first fully connected layer has an activation function; the dimensionality reduction process of the text feature data includes: inputting the text feature data into the text information compression model, and obtaining text compression features corresponding to the text feature data according to the output of the text information compression model.

[0018] In a third exemplary embodiment, the feature dimensionality reduction module includes a personalized information compression model; the personalized information compression model includes a third fully connected layer and a fourth fully connected layer; the number of neurons in the third fully connected layer matches the feature dimension of the personalized feature, and the third fully connected layer has an activation function; the dimensionality reduction process of the personalized feature includes: inputting the personalized feature into the personalized information compression model, and obtaining the personalized compression feature corresponding to the personalized feature according to the output of the personalized information compression model.

[0019] In a fourth exemplary embodiment, the adaptive alignment module includes multiple sub-time dimension alignment models, and the number of sub-time dimension alignment models is the same as the number of time series feature types; each sub-time dimension alignment model includes a first vector dimension rearrangement layer, an adaptive maximum pooling layer and a second vector dimension rearrangement layer.

[0020] In a fifth exemplary embodiment, the adaptive maximum pooling layer has a time step hyperparameter. During the training process of the emotion recognition model, the time step hyperparameter is continuously adjusted through the back propagation algorithm, and the target time step hyperparameter that meets the preset model verification performance is used as the time step hyperparameter for performing the emotion recognition task.

[0021] In a sixth exemplary embodiment, the time series hybrid attention module includes a time series data modeling layer, a multi-head attention calculation module and a hybrid time series output module; wherein, the time series data modeling layer includes a recurrent neural network for time series data modeling; the multi-head attention calculation module includes an implicit vector splicing layer, a fifth fully connected layer, a normalized exponential function layer, a score splicing layer and a multi-head output splicing layer; the hybrid time series output module includes a vector partitioning layer, a dimensionality reduction layer, a dimensionality reduction vector splicing layer and a vector normalization layer.

[0022] In the seventh exemplary embodiment, the model parameters of the multi-head attention calculation module include the number of heads. During the training of the emotion recognition model, the number of heads of the emotion recognition model is continuously adjusted through the back propagation algorithm, and the target number of heads that meets the preset model verification performance is used as the number of heads used to perform the emotion recognition task.

[0023] In an eighth exemplary embodiment, the time series data modeling layer is a time series feature mapping model constructed based on a long short-term memory neural network, and the time series feature mapping model includes multiple sub-time series feature mapping models, and the number of each sub-time series feature mapping model is the same as the number of time series feature types; the generation process of the implicit state data of each type of time series feature of the target type feature at each time step includes: for each type of time series feature, the mixed time series data of the previous time step, the current time series feature and the implicit state data and cell state data corresponding to the previous time step are respectively input into the corresponding current sub-time series feature mapping model, and according to the output of the current sub-time series feature mapping model, the implicit state data of the current time series feature at the current time step is obtained.

[0024] In a ninth exemplary embodiment, the feature dimensionality reduction module includes a time series mixed information compression model; the time series mixed information compression model includes a vector splicing layer, a sixth fully connected layer and a seventh fully connected layer; the number of neurons in the sixth fully connected layer matches the feature dimension of the vector splicing layer, and the sixth fully connected layer has an activation function; the dimensionality reduction process of the fused emotional response feature includes: inputting the mixed time series data of each time step and the implicit state data of each type of time series features into the time series mixed information compression model, the vector splicing layer splices the mixed time series data of the same time step and the implicit state data of each type of time series features, and passes the time series mixed splicing vector of the current time step through the sixth fully connected layer and the seventh fully connected layer to obtain the emotional response compression feature corresponding to the fused emotional response feature.

[0025] The present invention also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of the emotion recognition method as described in any of the preceding items when executing a computer program stored in a memory.

[0026] The present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the emotion recognition method as described in any of the preceding items are implemented.

[0027] Finally, the present invention also provides an emotion recognition system, including a first computing device and a second computing device, wherein the first computing device and the second computing device are connected via a target bus, and the second computing device has parallel computing capabilities; wherein the first computing device includes a human-computer interaction module and an emotion response data acquisition module, and when an emotion recognition task is received, the target object is displayed via the human-computer interaction module, and the emotion response data acquisition module is triggered to collect the user's emotion response data of the target object; a personality test questionnaire page is sent to a user terminal pre-bound to the user, and the personality test questionnaire data sent by the user terminal is received; the first computing device sends the emotion response data and the personality test questionnaire data to the second computing device via the target bus interface; the second computing device implements the steps of the emotion recognition method described in any of the previous items when executing the computer program stored in the memory.

[0028] In a first exemplary embodiment, the second computing device includes a data storage device, a data distribution module, a text information processing module, a personal information processing module, a tri-modal time series processing module and an emotion recognition module, and the data distribution module, the text information processing module, the personal information processing module, the tri-modal time series processing module and the emotion recognition module adopt an asynchronous communication method; wherein the data distribution module is configured to extract text data and time series data from the emotional response data, send the text data to the text information processing module, input the personality test questionnaire data into the personal information processing module, and input the time series data into the tri-modal time series processing module; the text information processing module is configured to extract text feature data from the text data, perform dimensionality reduction processing on the text feature data, and input text compression features into the emotion recognition module; the personal information processing module is configured to extract text feature data from the text data, perform dimensionality reduction processing on the text feature data, and input text compression features into the emotion recognition module; the personal information processing module is configured to extract text feature data from the text data, perform dimensionality reduction processing on the text feature data, and input text compression features into the emotion recognition module according to the personality The test questionnaire data generates personalized features, performs dimensionality reduction processing on the personalized features, and inputs the personalized compressed features into the emotion recognition module; the trimodal time series processing module is configured to extract target type features from the time series data, align various time series features of the target type features in the time dimension, and fuse various time series features at each time step based on the attention mechanism to obtain fused emotional response features, perform dimensionality reduction processing on the fused emotional response features, and input the emotional response compressed features into the emotion recognition module; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; the emotion recognition module is configured to splice the emotional response compressed features, the text compressed features, and the personalized compressed features into multimodal feature data, perform emotion recognition on the multimodal feature data, and display the emotion recognition results to the user through the target bus interface and the human-computer interaction module.

[0029] The advantage of the technical solution provided by the present invention is that personalized data is obtained through a personality test questionnaire, multi-dimensional emotional response features are extracted from the user's emotional response data, and personalized data and multi-dimensional emotional response features are used together as input features for emotion recognition. Since the emotion recognition result is obtained by integrating audio-visual, text and personalized information, and fully considering personalized factors, the emotion recognition accuracy can be effectively improved. Furthermore, the features of different time dimensions are fused after alignment, and the features of different dimensions complement each other, which is conducive to improving the accuracy and richness of feature representation, and further improving the accuracy of emotion recognition. In addition, by reducing the dimensionality of features of different dimensions, the speed of emotion recognition can be effectively improved, and efficient and high-accuracy emotion recognition can be achieved.

[0030] In addition, the present invention also provides a corresponding implementation system, electronic device, non-volatile storage medium and computer program product for the emotion recognition method, which further makes the method more practical. The system, electronic device, non-volatile storage medium and computer program product have corresponding advantages.

[0031] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of a flow chart of an emotion recognition method provided by the present invention; Figure 2 A schematic diagram of a flow chart of another emotion recognition method provided by the present invention; Figure 3 A schematic diagram of the model structure and data processing flow of the emotion recognition model provided by the present invention in an exemplary application scenario; Figure 4 A schematic diagram of the model structure and data processing flow of the adaptive alignment module provided by the present invention in an exemplary application scenario; Figure 5 A schematic diagram of the model structure and data processing flow of the temporal hybrid attention module provided by the present invention in an exemplary application scenario; Figure 6 A schematic diagram of the model structure of the time series feature mapping model provided by the present invention in an exemplary application scenario and the data processing flow under a single time step; Figure 7 A schematic diagram of a structural framework of an exemplary embodiment of the emotion recognition device provided by the present invention; Figure 8 A schematic diagram of a structural framework of an exemplary embodiment of the emotion recognition device provided by the present invention; Fig. 9 A schematic diagram of a structural framework of an exemplary embodiment of an electronic device provided by the present invention; Fig.10 A structural diagram of an exemplary implementation of the emotion recognition system provided by the present invention; Fig.11 A schematic diagram of the structural framework of another exemplary embodiment of the emotion recognition system provided by the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0035] With the development of artificial intelligence technology, in order to enhance the effectiveness of emotion recognition, emotion recognition has evolved from single-modal recognition to multi-modal emotion recognition. That is, when performing emotion recognition tasks, any combination of visual features, text features, and auditory features will be integrated as input features for emotion recognition, such as health diagnosis tasks, public sentiment analysis tasks, educational classroom environment monitoring tasks, and emotion-aware recommendation systems.

[0036] It is understandable that different individuals may show different emotional responses to the same stimulus based on their unique personality traits, that is, emotional responses are subjective. In order to further improve the accuracy of emotion recognition and enhance the applicability of emotion recognition results in different and subtle user interactions, the inherent specificity of individual responses to stimuli requires a personalized approach in the emotion recognition system. According to the modality involved, it can be divided into unimodal personalized emotion recognition and multimodal personalized emotion recognition. Unimodal personalized emotion recognition is to extract individual-specific clues from text data or visual data to facilitate emotion recognition. Multimodal personalized emotion recognition integrates different information modes and utilizes the specific attributes of individuals to develop methods suitable for individual emotion analysis. To achieve personalized emotion recognition, the emotion dataset used by the emotion recognition model needs to be carefully annotated from two dimensions: the assembly of the emotion dataset and the characterization of personality. Among them, the assembly of the emotion dataset can be completed by performance or induction. The performance method is to perform a certain category of emotional response. Although it can provide a clear classification name, it may not accurately reflect the range of real emotional responses; the induction method is more in line with the real emotional experience of individuals. Personality traits are multifaceted and can significantly affect the emotional response to stimuli.

[0037] When the related art performs emotion recognition based on personalized factors, the user's physiological data, such as EEG data, will be used as the user's unique individual characteristics. However, personalized emotion recognition based on physiological signals cannot accurately identify the user's emotional state. In view of this, the present invention performs feature extraction according to different input modalities to obtain the initial feature representation of each modality, and then synchronizes the time features across modalities. The time-synchronized features are processed by a temporal hybrid attention mechanism, which promotes the dynamic and interactive fusion of multimodal information. Finally, the various features obtained are subjected to dimensionality reduction processing, and these features are fused to finally form a comprehensive feature set, which is used as the input of emotion recognition, which can solve the problem of low recognition accuracy caused by insufficient consideration of personalized factors in the related art. After introducing the technical scheme of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In some other examples, the methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0038] First see Figure 1 , Figure 1 A flowchart of an emotion recognition method provided in this embodiment may include the following contents: S101: Extracting text feature data and target type features from the emotional response data of the user perceiving the target object.

[0039] Among them, the user is a target with emotion recognition needs, such as a person or an organism with emotional response ability, the target object is an object that can cause the user's emotional response or is used for emotional response testing, including but not limited to video, audio, and organisms, and the emotional response data is the information recording the user's emotional response in the process of perceiving the target object, which can be recorded in any electronic data format, such as video files, audio files, text files, or any combination, and the perception action can be, for example, watching, touching, listening, or any combination. Exemplarily, the emotional response data can be the data of a person's viewing record of a video, which can reflect the person's emotional response to watching a specific video. In order to analyze the emotional response data, it is necessary to extract the user's emotional response features. This embodiment includes at least text feature data and target type features. The text feature data is text features extracted from the emotional response data that use text to express emotional responses. The target type features include at least two types of timing features among audio timing features, face timing features, and video timing features, namely, audio features extracted from the emotional response data that use sound to express emotional responses, facial expression features extracted from the emotional response data that use facial expressions to express emotional responses, and video frame features extracted from the emotional response data that use both vision and hearing to express emotional responses. These features all have time attributes. For the sake of ease of description, this embodiment defines them as audio timing features, face timing features, and video timing features. When extracting text feature data and target type feature audio timing features, face timing features, and video timing features from the emotional response data, any method capable of extracting features recorded in the relevant technology can be used. For example, when the emotional response data is video data, the emotional response data is input into a pre-trained audio model to obtain a vector expression of the audio as an audio timing feature; it is input into a pre-trained face model to perform a vector expression of the face to obtain a face timing feature; the emotional response data is feature extracted using a pre-trained visual model to obtain a vector expression of the video frame as a video timing feature. In order to facilitate text feature extraction, the audio data carried by the emotional response data can be converted into text data, and then a pre-trained language model is used to perform a vector expression of the text from the text data to obtain text feature data.

[0040] S102: Generate personalized features based on the user's personality test questionnaire data.

[0041] This embodiment obtains the personalized information of the user through the personality test questionnaire data. The personality test questionnaire data is the user's personality data collected when the user is performing emotion recognition. Compared with physiological data, it can more accurately describe the user's personality or individuality. Compared with analyzing based on emotional history records and personalized consultation files, it can better fit the user's current emotions and more accurately reflect the user's current personality characteristics. Since the data format of the personality test questionnaire data may be inconsistent with the data format that the computer system can process, it needs to be converted into feature data that can be used for subsequent data processing. Those skilled in the art can process the personality test questionnaire data accordingly according to the data format required by the computer system, and the processed data is the personalized feature.

[0042] S103: Align various time series features of the target type features in the time dimension, and fuse various time series features at each time step based on the attention mechanism to obtain a fused emotional response feature.

[0043] It can be understood that the text feature data represents the expression of the text contained in the entire emotional response data. This type of feature data has no time dimension, while the target type data, such as video time series data, is a frame-based expression of the video, the face time series feature is also based on frame sampling, and the audio time series feature is based on the time series. The spatial window obtains the feature, so the three types of time series features of the target type feature are not corresponding in the time dimension, so it is necessary to unify the three types of time series features in the time dimension. After the three types of time series features are aligned in the time dimension, in order to fully integrate the three types of time series features, the three types of time series features at different time steps will be fused based on the attention mechanism, so that the data under different modes can be complementary in information. For the sake of convenience of description, it is defined as a fusion of emotional response features. Among them, any method of obtaining information between different time steps can be used to obtain the three types of time series features at different time steps, and the present invention does not make any limitation to this.

[0044] S104: Perform dimensionality reduction processing on the fused emotional response features, text feature data and personalized features to obtain multimodal feature data for emotion recognition.

[0045] After the above steps obtain the fused emotional response features, text feature data and personalized features, in order to achieve efficient and high-precision emotion recognition, the fused emotional response features, text feature data and personalized features can be reduced in dimension, and the various compressed features obtained after the dimensionality reduction can be used as the input of any neural network model that can realize emotion recognition, and the corresponding emotion recognition results can be obtained through the neural network model.

[0046] In the technical solution provided in this embodiment, personalized data is obtained through a personality test questionnaire, multidimensional emotional response features are extracted from the user's emotional response data, and the personalized data and multidimensional emotional response features are used together as input features for emotion recognition. Since the emotion recognition result is obtained by integrating audio-visual, textual and personalized information, and fully considering personalized factors, the emotion recognition accuracy can be effectively improved. Furthermore, the features of different time dimensions are fused after alignment, and the features of different dimensions complement each other, which is conducive to improving the accuracy and richness of feature representation, and further improving the accuracy of emotion recognition. In addition, by reducing the dimensionality of features of different dimensions, the speed of emotion recognition can be effectively improved, and efficient and high-accuracy emotion recognition can be achieved.

[0047] In the above embodiment, there is no limitation on how to align various types of time series features of target type features in the time dimension. This embodiment provides an implementation method for unifying different types of time series features in target type features in the same time dimension, which may include the following contents: The feature dimensions of various types of time series features of the target type features are rearranged to replace the second dimension and the third dimension of the feature vectors corresponding to the various types of time series features; for the various types of time series features after the dimension replacement, an adaptive maximum pooling operation is performed on the new third dimension to output various types of time series features with the same third dimension; according to the dimensional output format of the target type features, the feature dimensions of various types of time series features are rearranged again.

[0048] It is understandable that when extracting audio timing features from audio data contained in emotional response data, each audio data is usually divided into different segments for feature extraction, and each segment has a corresponding feature dimension according to the feature extraction method. In this way, the three dimensions of the vector representation corresponding to the audio timing feature can be obtained according to the total number of segments, the feature dimension of each segment, and the number of audio data. When extracting video timing features from emotional response data, a single frame of video has a feature dimension according to the extraction method, and a video frame has a frame number. According to the total number of videos, the number of video frames, and the feature dimension of a single video frame, the three dimensions of the vector representation corresponding to the video timing feature can be obtained; the number of frames of a face is the same as the number of video frames, and the feature dimension of the face feature has a corresponding face feature dimension according to different face feature extraction methods. According to the total number of face images, the number of face frames, and the face feature dimension, the three dimensions of the vector representation corresponding to the face timing feature can be obtained. In this embodiment, any operation or method that can realize the rearrangement of tensor dimensions or vector dimensions can be used to rearrange the dimensions of various types of timing features, for example, it can be achieved through a permute operation. For example, the number of audios is N, the number of audio clips is 102, and the audio feature dimension is 5120. The three dimensions of the vector representation corresponding to the audio timing feature are N×102×5120. After rearranging the feature dimensions of the audio timing feature, the new three dimensions of the audio timing feature are N×5120×102. In this way, Adaptive Max Pool can be used to perform parameter-free adaptive pooling operations on a one-dimensional dimension. Regardless of how the input size changes, the adaptive maximum pooling operation can output a tensor of a fixed size. For example, Adaptive Max Pool 1d can convert a 4×3×7 input into a 4×3×3 output, and Adaptive Max Pool 2d can convert a 2×64×8×9 input into a 2×64×3×6 output. The time step of each timing feature is unified to the specified time step hyperparameter. This unification method is more direct than the traditional method of filling with 0, and the time step hyperparameter can be used as an adjustable hyperparameter to control the parameter scale of the entire network. After unifying the time steps of various time series features, in order to meet the input format requirements of subsequent data processing, the feature dimensions of various time series features can be rearranged again.

[0049] Exemplarily, in order to improve the use of emotion recognition tasks in hardware environments with different resources and scenarios with different emotion recognition accuracies, the present invention also provides an adjustment of the size of the network used for emotion recognition tasks. Users can adjust the time step hyperparameter according to the actual application scenario and issue an instruction carrying the new time step hyperparameter. For ease of description, the instruction can be defined as a network scale adjustment instruction. When the network scale adjustment instruction is received, the network scale adjustment instruction is parsed according to the format of the network scale adjustment instruction to obtain the time step hyperparameter required by the current user; based on the time step hyperparameter, an adaptive maximum pooling operation is performed on various types of time series features after the permutation of dimensions to obtain various types of time series features whose third dimensions are all time step hyperparameters.

[0050] From the above, it can be seen that this embodiment realizes the time alignment of each time series feature through adaptive maximum pooling, and controls the network scale by setting an optional alignment length, thereby improving the use of emotion recognition tasks in hardware environments with different resources and being suitable for scenarios with different emotion recognition accuracies.

[0051] In order to further improve user privacy security, based on the above embodiment, the present invention also implements the generation process of personality test questionnaire data on a user-controllable device, which may include the following contents: When a personalized data collection request is detected, the personality test questionnaire page is sent and displayed on the user terminal pre-bound by the user; when it is detected that the data is completed, the personality test result is generated according to the personality test data on the personality test questionnaire page; when a test result sending instruction is received, the personality test result is encrypted and transmitted, and the received personality test result is converted into personalized features according to the personalized data input format.

[0052] Among them, the user terminal can be, for example, a personal computer, a smart phone, or a tablet of the user, and the personalized collection request can be triggered by scanning a QR code. Correspondingly, the personality test questionnaire page can be the result of scanning the QR code. Of course, the personality test questionnaire page can also be sent to the user terminal by mobile phone number, email, etc., which does not affect the implementation of the present invention. The personality test result can be the personality name of the personality type to which the user belongs, or it can be the personality test score of the user. Any encryption technology can be used to encrypt the personality test result first, and then the encrypted data is transmitted to protect the user's privacy.

[0053] As can be seen from the above, this embodiment collects personality test data on a user-controllable device and transmits it in encrypted form, which can ensure the security of user privacy data.

[0054] The above embodiment does not impose any limitation on the type of personality test questionnaire data. Based on the above embodiment, the present invention also provides an exemplary method for generating a personality test questionnaire page, which may include the following contents: Pre-build a personality test questionnaire table, and set the display trigger condition of the personality test questionnaire page to a personalized data collection request; obtain personality test text data including Big Five personality test questions and scores of each question, and fill in the personality test questionnaire table based on the personality test text data to generate a personality test questionnaire page.

[0055] Among them, the personality test questionnaire table includes the first personality option, the second personality option, the third personality option, the fourth personality option and the fifth personality option; the first personality option, the second personality option, the third personality option, the fourth personality option and the fifth personality option all have personality scores, the first personality aspect, the second personality aspect and the third personality aspect; each first personality aspect, each second personality aspect and each third personality aspect has a corresponding personality aspect score. The first personality option, the second personality option, the third personality option, the fourth personality option and the fifth personality option can correspond to the names of each personality of the Big Five Personalities. For example, the Big Five Personality Test Questions and the scores of each question can select 60 questions of the second edition of the Five Personality Questionnaire, and each of the 60 questions has a corresponding score. Based on these questions, the 5 scores of the Big Five Personalities on the personality test questionnaire table and the 15 scores of the corresponding aspects are scored. In order to make full use of this information, all data obtained from the personality test questionnaire table can be spliced ​​together into an 80-dimensional vector. Since the numerical range of the Big Five Personalities and the numerical range of the original questions are quite different, all personality traits can be normalized to between 0 and 1 using a normalization method.

[0056] As can be seen from the above, this embodiment uses the Big Five personality scale to describe personalized information, which is conducive to improving the expression accuracy of personalized data. Integrating this personalized feature in multimodal emotion recognition can effectively improve the accuracy of emotion recognition.

[0057] The above embodiment does not limit how to fuse the time series features of the target type features. This embodiment also provides an exemplary implementation method for fusing various types of time series features at each time step based on the attention mechanism, which may include the following contents: Obtain the implicit state data of various types of time series features of the target type features at each time step; perform multi-head attention calculation on the current implicit state data of the current time step to obtain the multi-head weighted vectors corresponding to various types of time series features; based on the multi-head weighted vectors corresponding to various types of time series features, perform dimensionality reduction processing on various types of time series features, normalize the concatenated vectors of various types of time series features after dimensionality reduction, and obtain the mixed time series data of the current time step; obtain the fused emotional response features based on the mixed time series features of each time step.

[0058] In this embodiment, the implicit state data of various time series features at each time step can be realized by any recurrent neural network for modeling time series data, that is, the information between different time steps can be modeled by cyclically inputting the data of different time steps into the memory structure of the same structure. Whenever the implicit state data of various time series features at a time step is obtained, a multi-head attention calculation is performed on it. Since various time series features need to go through the attention mechanism, in order to enhance the parallelism of the calculation, the three vectors can be spliced ​​together to obtain a long splicing vector using the vector splicing operation, that is, the current implicit state data is spliced ​​into an implicit state splicing vector. In order to perform multi-head attention calculation, the implicit state splicing vector needs to be multi-head copied according to the number of heads; each head independently and in parallel calculates the attention score: the implicit state splicing vector is pre-calculated with multiple heads, and the multi-head weight pre-calculation can be implemented, for example, by a full connection method to obtain a weight pre-vector. In order to calculate the specific score, it is necessary to pass the activation function, so it is divided according to the original length. For the convenience of description, the implicit state splicing vector obtained by the division is defined as a sub-implicit state splicing vector, and the implicit state splicing vector is divided into multiple sub-implicit state splicing vectors corresponding to the number of heads based on the feature dimension of each current implicit state data; for each head based on the weighted pre-vector, the attention score of each sub-implicit state splicing vector is calculated separately, and the attention scores are spliced ​​to obtain the total attention score. After the attention scores are spliced ​​together, for the convenience of description, it can be defined as the total attention score.

[0059] Among them, the multi-head attention mechanism divides the query, key and value matrices into multiple heads, that is, multiple subspaces, and each head has different linear transformation parameters. Each head independently calculates the attention score and generates an attention-weighted output. In the multi-head attention mechanism, the attention weight calculation of each head can be achieved by scaling the dot product attention. The dot product of the query matrix and the key matrix is ​​calculated. After scaling and adding the bias, the softmax (normalized exponential function) is used to obtain the attention weight. The value vector is weighted and summed using the attention weight, and the attention weighted output is obtained. The attention weighted outputs of each head are spliced ​​to obtain the weighted total vector. Of course, after the attention scores are spliced ​​together, the original spliced ​​vector can be repeated multiple times to obtain the total vector, and the weighted vector is obtained by multiplying the corresponding elements. The weighted total vectors are divided multiple times in the order of various types of time series features to obtain multi-head weighted vectors corresponding to various types of time series features. For example, the weighted vectors can be divided into various types of multi-head weighted vectors in the order of audio time series features, face time series features, and video time series features. Then, after passing through three dimensionality reduction structures, such as audio dimensionality reduction structure, face dimensionality reduction structure, and video dimensionality reduction structure, the reduced dimensionality vectors of various types of time series features are obtained. The spliced ​​vector of the output end is obtained by splicing the vectors of various types of time series features. Finally, softmax can be used for normalization to obtain the output of the current time step.

[0060] As can be seen from the above, this embodiment uses a parallel approach to accelerate calculations during the temporal hybrid attention process, effectively improving the execution efficiency of the emotion recognition task. In addition, the number of heads can be a configurable hyperparameter to facilitate control of the execution efficiency of the entire task. The method of splicing first and then calculating in the multi-head attention calculation process can enhance parallelism and further improve the output efficiency of the emotion recognition results.

[0061] Based on the above embodiments, the present invention also provides another implementation method of the emotion recognition method from the perspective of the model performing the emotion recognition task, see Figure 2 , Figure 2 A flow chart of another emotion recognition method provided by the present invention may include the following contents: S201: Obtain an emotion recognition training sample set and build an emotion recognition model.

[0062] In this embodiment, the emotion recognition training sample set is a training sample set for training an emotion recognition model. The emotion recognition training sample set may include a plurality of emotion sample data carrying emotion labels. The emotion sample data is the emotion response data of the user perceiving the target object and the personality test questionnaire data corresponding to the user. For example, the emotion response data is an emotion response video, and the personality test questionnaire data is a Big Five personality test score. The emotion sample data is an emotion response video and a corresponding test score, and the labels are sadness, happiness, relaxation, surprise, fear, disgust, anger, neutrality, etc. The emotion recognition model includes an input layer, a multimodal feature processing sub-model and an emotion recognition sub-model. The input layer receives the training sample data in the training process of the input emotion recognition model and the reasoning task data in the reasoning process. The multimodal feature processing sub-model performs feature extraction, alignment and fusion of time series feature data, and multimodal feature dimensionality reduction processing on the training sample data and reasoning task data input through the input layer to obtain multimodal feature data, which may include a feature extraction layer, an adaptive alignment module, a time series mixed attention module and a feature dimensionality reduction module; the emotion recognition sub-model performs classification and recognition according to the multimodal feature data output by the multimodal feature processing sub-model. It can concatenate all input feature vectors and then use a fully connected network structure to map them to the dimensions of the emotion output space, perform classification and recognition through the constructed classification task, and finally output the corresponding emotion recognition result.

[0063] S202: Using the emotion recognition training sample set, the emotion recognition model is cyclically trained based on the target classification loss function until a preset model training stop condition is reached.

[0064] The target classification loss function can adopt any loss function used to train the neural network model for classification and recognition tasks. For the sake of distinction, the adopted loss function is defined as the target classification loss function. For example, the cross entropy loss function can be used to train the weight parameters of each module in the entire network. The preset model training stop conditions may be reaching a preset number of iterations, the model performance reaching preset performance requirements, and the model converging. Those skilled in the art may train the emotion recognition model of S201 based on the emotion recognition training sample set of step S201 according to any model training method recorded in the relevant technology, which does not affect the implementation of the present invention.

[0065] S203: Input the user's emotional response data of the target object and the user's personality test questionnaire data into the trained emotion recognition model, and obtain the emotion recognition result according to the output of the emotion recognition model.

[0066] This step refers to that when executing the user's emotion recognition task, the user's emotional response data of the target object and the user's personality test questionnaire data are used as reasoning task data and input into the trained emotion recognition model. The input layer of the emotion recognition model is used to receive the user's emotional response data of the target object and the user's personality test questionnaire data; the feature extraction layer extracts text feature data and target type features from the emotional response data, and generates personalized features according to the user's personality test questionnaire data; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; the adaptive alignment module aligns various types of time series features of the target type features in the time dimension; the time series mixed attention module fuses various types of time series features at each time step based on the attention mechanism to obtain a fused emotional response feature; the feature dimensionality reduction module performs dimensionality reduction processing on the fused emotional response features, text feature data and personalized features; the emotion recognition sub-model splices the emotional response compression features, text compression features and personality compression features into multimodal feature data, performs emotion recognition on the multimodal feature data, and outputs the emotion recognition result.

[0067] As can be seen from the above, the emotion recognition model of this embodiment is based on the personality test questionnaire data as personalized data, and at the same time, it is input into the emotion recognition model together with the multi-dimensional emotion response features extracted from the user's emotion response data. The emotion recognition model can learn more accurate emotion recognition capabilities from fully considering personalized factors and integrating audio-visual, text and personalized information, effectively improving the execution accuracy of the emotion recognition task. Furthermore, the features of different time dimensions are fused after alignment, and the features of different dimensions complement each other, which is conducive to improving the accuracy and richness of the input feature representation of the emotion recognition model, and further improving the execution accuracy of the emotion recognition task. In addition, by reducing the dimensionality of features of different dimensions, the data processing efficiency of the emotion recognition model can be accelerated, and efficient and high-accuracy emotion recognition can be achieved.

[0068] The above embodiment does not limit how the feature extraction layer of the emotion recognition model extracts various modal features. Based on the above embodiment, the present invention also provides an exemplary implementation of the feature extraction layer, which may include the following contents: The feature extraction layer includes one or more of an audio feature extraction model, a face recognition model, a visual feature extraction model, and a language feature extraction model. Whether the audio feature extraction model, the face recognition model, and the visual feature extraction model are all included is matched according to the type of temporal features contained in the target type features. The process of extracting text feature data and target type features from the emotional response data may include: inputting the emotional response video data into the audio feature extraction model, and obtaining audio temporal features according to the output of the audio feature extraction model; inputting the emotional response video data into the face recognition model, and performing face recognition and face feature extraction according to the face recognition model to obtain face temporal features; inputting the emotional response video data into the visual feature extraction model, and obtaining video temporal features according to the output of the visual feature extraction model; converting the audio data of the emotional response video data into text data, and inputting the text data into the language feature extraction model, and obtaining text feature data according to the output of the language feature extraction model.

[0069] In this embodiment, the audio feature extraction model, the face recognition model, the visual feature extraction model, and the language feature extraction model can adopt a neural network model that has been pre-trained and can realize the corresponding functions. For example, the audio feature extraction model can adopt a pre-trained wavLM-base model (a neural network model for extracting audio features), the face recognition model can adopt a pre-trained FaceNet (face detection network model) model, the visual feature extraction model can adopt a pre-trained ViT-B / 16 (visual network model) model, and the language feature extraction model can adopt a Transformer (transformer network model) model. For the emotional response video data, audio extraction can be performed first, and then the face in the emotional response video data can be detected and extracted, and then the face pre-trained model can be used to perform vector expression of the face; the visual network model can be used to extract features from the video frame of the emotional response video data to obtain a vector expression of the video frame; for the text converted from the audio, the pre-trained language model can be used to perform vector expression of the text.

[0070] The above embodiment does not limit how to implement feature dimensionality reduction for multiple modal features. Based on the above embodiment, the present invention also provides an exemplary implementation of a feature dimensionality reduction module, which may include the following contents: like Figure 3As shown, the feature dimension reduction module may include a text information compression model, a personality information compression model, and a time series mixed information compression model. Among them, the text information compression model includes a first fully connected layer and a second fully connected layer; the number of neurons in the first fully connected layer matches the feature dimension of the text feature data, and the first fully connected layer has an activation function; the personality information compression model includes a third fully connected layer and a fourth fully connected layer; the number of neurons in the third fully connected layer matches the feature dimension of the personalized feature, and the third fully connected layer has an activation function; the time series mixed information compression model includes a vector splicing layer, a sixth fully connected layer, and a seventh fully connected layer; the number of neurons in the sixth fully connected layer matches the feature dimension of the vector splicing layer, and the sixth fully connected layer has an activation function. Among them, the activation function can be flexibly selected according to actual conditions, for example, the ReLU (Linear rectification function, rectified linear unit) activation function can be used. The compression models are similar in network structure. In order to adapt to the feature dimensions of different structures, the number of neurons in the first fully connected layer can be changed. The activation function is only applied to the first layer of the fully connected network, and the second layer directly uses the linear transformation method to maintain the structure of the data.

[0071] Based on the above-mentioned feature dimension reduction module, the dimension reduction process of text feature data may include: inputting the text feature data into the text information compression model, and obtaining the text compression features corresponding to the text feature data according to the output of the text information compression model. The dimension reduction process of personalized features may include: inputting the personalized features into the personalized information compression model, and obtaining the personalized compression features corresponding to the personalized features according to the output of the personalized information compression model. The dimension reduction process of fused emotional response features may include: inputting the mixed time series data of each time step and the implicit state data of various time series features into the time series mixed information compression model, the vector splicing layer splices the mixed time series data of the same time step and the implicit state data of various time series features, and the time series mixed splicing vector of the current time step is passed through the sixth fully connected layer and the seventh fully connected layer to obtain the emotional response compression features corresponding to the fused emotional response features.

[0072] It can be seen from the above that, in this embodiment, when reducing the dimensionality of various modal features, nonlinear changes are only performed in the first fully connected layer, and the second fully connected layer maintains the data structure through linear changes, which is conducive to improving the execution efficiency of the emotional task.

[0073] The above embodiment does not impose any limitation on the network structure of the adaptive alignment module. This embodiment also provides an exemplary implementation method, which may include the following contents: The adaptive alignment module includes multiple sub-time dimension alignment models, and the number of each sub-time dimension alignment model is the same as the number of time series feature types; each sub-time dimension alignment model includes a first vector dimension rearrangement layer, an adaptive maximum pooling layer, and a second vector dimension rearrangement layer. The first vector dimension rearrangement layer and the second vector dimension rearrangement layer can adopt the same operation mode, such as both can adopt the permute operation, of course, they can also be different, which does not affect the implementation of the present invention. Figure 4 For example, for the original audio file, it needs to be divided into different segments for feature extraction. The pre-trained model wavLM-base is used for feature extraction. The feature dimension of each segment is 5120. The number of segments is related to the length of the audio. Figure 3 Take 102 audio clips as an example. The video time series features are extracted using ViT-B / 16, with a single frame feature dimension of 512, and the number of frames in the figure is 78; the number of frames of the face is the same as the video, but the feature extraction uses the FaceNet pre-trained model, with a feature dimension of 80. First, the permute operation is used to change the middle time series dimension from the second dimension to the third dimension, so that the AdaptiveMaxPool1d operation can be used for parameter-free adaptive pooling operations. This pooling operation unifies the time step to 64. After unification, the permute operation is used again to make the data format meet the input format requirements of the later modules.

[0074] Furthermore, the adaptive maximum pooling layer has a time step hyperparameter. During the training process of the emotion recognition model, the time step hyperparameter is continuously adjusted through the back propagation algorithm, and the target time step hyperparameter that meets the preset model verification performance is used as the time step hyperparameter for performing the emotion recognition task.

[0075] As can be seen from the above, this embodiment realizes the time alignment of each time series feature through the adaptive alignment module, and realizes the control of the network scale by setting the selectable time step hyperparameter.

[0076] The above embodiment does not impose any limitation on the network model structure of the temporal hybrid attention module. Based on the above embodiment, the present invention also provides an exemplary model structure of the temporal hybrid attention module, which may include the following contents: The time series hybrid attention module includes a time series data modeling layer, a multi-head attention calculation module and a hybrid time series output module; wherein, the time series data modeling layer includes a recurrent neural network for time series data modeling; the multi-head attention calculation module includes an implicit vector splicing layer, a fifth fully connected layer, a normalized exponential function layer, a score splicing layer and a multi-head output splicing layer; the hybrid time series output module includes a vector partitioning layer, a dimensionality reduction layer, a dimensionality reduction vector splicing layer and a vector normalization layer.

[0077] In this embodiment, the time series data modeling layer can adopt any recurrent neural network structure. After being processed by the adaptive alignment module, various time series features have been aligned in time steps, so the step lengths of the time series data modeling layer are completely consistent, avoiding inconsistent filling of time series. Figure 5 As shown in the figure, the implicit state vectors obtained by the various time series features output by the adaptive alignment module after the time series data modeling layer can be defined as audio implicit state vectors, face implicit state vectors and video implicit state vectors. After obtaining the output of the current step, since various time series features need to go through the attention mechanism, in order to enhance the parallelism of the calculation, the three vectors are first concatenated together using the vector concatenation operation to obtain a long concatenated vector. In order to perform multi-head attention calculations, multi-head replication is performed. Figure 5 Taking 3 heads as an example, after full connection, the weighted forward vector is obtained. In order to calculate the attention score, an activation function is required, so it can be divided according to the original length, and three attention scores can be obtained after softmax. After splicing the attention scores, the original spliced ​​vector is repeated many times to obtain the total vector, and the weighted vector is obtained by multiplying the corresponding elements. Finally, the weighted vector is divided into various types of multi-head weighted vectors in the order of three different modes, and then the corresponding dimensionality reduction structures such as audio dimensionality reduction, face dimensionality reduction, and video dimensionality reduction are obtained to obtain the dimensionality reduction vectors corresponding to various types of time series features. The spliced ​​vector at the output end is obtained by splicing the vectors corresponding to various types of time series features. After normalization using softmax, the output of the current time step is obtained. For ease of description, the mixed normalized vector between this time step can be marked with z.

[0078] Furthermore, the number of heads can be a variable hyperparameter, that is, the model parameters of the multi-head attention calculation module include the number of heads. During the training process of the emotion recognition model, the number of heads of the emotion recognition model is continuously adjusted through the back propagation algorithm, and the target number of heads that meets the preset model verification performance is used as the number of heads used to perform the emotion recognition task.

[0079] The above embodiment does not impose any limitation on the recurrent neural network model. This embodiment also provides an exemplary implementation method, which may include the following contents: The time series data modeling layer is a time series feature mapping model constructed based on a long short-term memory neural network. The time series feature mapping model includes multiple sub-time series feature mapping models, and the number of each sub-time series feature mapping model is the same as the number of time series feature types. The generation process of the implicit state data of each type of time series feature of the target type feature at each time step includes: for each type of time series feature, the mixed time series data of the previous time step, the current time series feature and its corresponding implicit state data and cell state data at the previous time step are respectively input into the corresponding current sub-time series feature mapping model, and the implicit state data of the current time series feature at the current time step is obtained according to the output of the current sub-time series feature mapping model.

[0080] Among them, the long short-term memory neural network is a single memory structure, which is a set of operations. The set of operations receives input data and generates intermediate output variables. In the long short-term memory neural network, the output intermediate variables are called implicit states and cell states. The mapping vector of each time series feature is modeled using a long short-term memory neural network. For the sake of ease of description, it is defined as a sub-time series feature mapping model. Taking the audio time series type as an example, the processing process of the corresponding sub-time series feature mapping model can be: the audio in the emotional response data is divided into L segments, and the output of each segment after feature extraction is an audio feature vector , where id represents the number of audio segments, ranging from 1 to L, and t represents the expression of the speech audio corresponding to the audio feature vector. The dimension is the same as that of the adaptive alignment module, such as 5120. The audio feature vector is the input of the corresponding sub-series feature mapping model. The sub-series feature mapping model contains three gating units, each of which is used to control the flow of information. The three gating units are input gate , the Forget Gate and output gate The function of the input gate is to input the audio feature vector and the implicit state vector of the previous time step The information of can be controlled, and the data processing of the input gate can be realized through the following relationship (1). The function of the forget gate is to restore the cell state vector of the previous time step The information flow is controlled, and the data processing of the forget gate can be realized by the following relationship (2). The output gate controls the amount of information from the output vector of the input gate and the forget gate that flows to the next hidden state, and the data processing of the input gate can be realized by the following relationship (3). The output of each gating unit is a vector of the same length as the input. The value range of each value in the vector is 0 to 1. 0 represents the shielding of the information at that position, 1 represents the full passage of the information at that position, and the intermediate value represents different degrees of control over the information at that position. Because the memory structure of the sub-series feature mapping model is exactly the same, this structure not only includes the calculation method, but also includes the weights of the calculation matrix. In order to maintain the unity of form, two vectors need to be constructed: the hidden state vector and the cell state vector , the dimensions of these two vectors are integers and Indicates, D indicates dimension, h indicates implicit state vector, and c indicates cell state vector. The dimension values ​​of these two vectors are the same. Among them, an intermediate variable of cell state The calculation process can be realized by the following relationship (4), the cell state vector of the idth audio clip is This can be achieved by the following relation (5): the implicit state vector of the idth audio clip is This can be achieved through the following relationship (6), and the implicit state vector is continuously updated in the above way and the cell state vector , the implicit state vector of each time step is generally used to represent the output feature vector of the current LSTM memory structure: ; (1) ; (2) ; (3) ; (4) ; (5) ; (6)

[0081] in, represents the multiplication of a matrix and a vector, represents the multiplication of corresponding elements, , , , Representatives The matrix for dimension mapping, the x in each matrix subscript represents the dimensional mapping of the audio feature vector, the f, i, and o in each matrix subscript correspond to the input gate, forget gate, and output gate, respectively, and c corresponds to the cell state vector. The dimension of the matrix is , m represents the value of the matrix dimension. , , , Representatives The matrix for dimension mapping is , the h in each matrix subscript represents the dimensional mapping of the implicit state vector, the f, i, and o in each matrix subscript correspond to the input gate, forget gate, and output gate, respectively, and c corresponds to the cell state vector. Represents the bias vector, which is the parameter that the network needs to learn. represents the sigmoid function, which can be expressed as the following relation (7), represents nonlinear mapping, x represents the audio feature vector, and e represents the exponent, which can be expressed as the following relation (8): ; (7)

[0082] . (8)

[0083] In order to further improve the extraction of effective information and the accuracy of emotion recognition, the data processing process based on the above-mentioned long short-term memory network is as follows: Figure 6 As shown, this embodiment also uses z as the output, which will play the role of input for calculating various gates and cell unit intermediate variables in neurons. Through the output vector of the temporal mixed attention module, three different sub-temporal feature mapping models can effectively and fully interact with each other. In each sub-temporal feature mapping model, z is used as part of the unit input, so that the mixed information can forward fuse the data information of each modality, and the characteristics of the sub-temporal feature mapping model on the time step are combined. With the passage of time, effective information is extracted from the perspective of high-level rich semantic information, thereby effectively improving the accuracy of emotion recognition.

[0084] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1-Figure 2 This is just a schematic and does not mean that this is the only execution order.

[0085] The present invention also provides a corresponding device for the emotion recognition method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The emotion recognition device provided by the present invention is introduced below. The device is used to implement the emotion recognition method provided by the present invention. In this embodiment, the emotion recognition device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the emotion recognition method disclosed in Example 1. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the emotion recognition device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The emotion recognition device described below and the emotion recognition method described above can be referenced to each other.

[0086] From the perspective of functional modules, see Figure 7 , Figure 7 The structure diagram of the emotion recognition device provided in this embodiment under a specific implementation manner, the device may include: The multimodal feature acquisition module 701 is used to extract text feature data and target type features from the emotional response data of the user's perception of the target object; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; and generate personalized features based on the user's personality test questionnaire data.

[0087] The time series feature processing module 702 is used to align various time series features of the target type features in the time dimension, and fuse various time series features at each time step based on the attention mechanism to obtain a fused emotional response feature.

[0088] The multimodal feature dimension reduction module 703 is used to perform dimension reduction processing on the fused emotional response features, text feature data and personalized features to obtain multimodal feature data for emotion recognition.

[0089] Exemplarily, in some implementations of the present embodiment, the above-mentioned time series feature processing module 702 can also be used to: rearrange the feature dimensions of various types of time series features of the target type features to replace the second dimension and the third dimension of the feature vectors corresponding to each type of time series features; perform an adaptive maximum pooling operation on the new third dimension for each type of time series features after the dimension replacement, and output various types of time series features with the same third dimension; and rearrange the feature dimensions of each type of time series features again according to the dimensional output format of the target type features.

[0090] As an exemplary implementation method of the above embodiment, the above-mentioned timing feature processing module 702 can be further used for: when receiving a network scale adjustment instruction, parsing the network scale adjustment instruction to obtain a time step hyperparameter; based on the time step hyperparameter, performing an adaptive maximum pooling operation on various timing features after the permutation of dimensions, to obtain various timing features whose third dimension is the time step hyperparameter.

[0091] Illustratively, in some other implementations of this embodiment, the multimodal feature acquisition module 701 may also be used to: when a personalized data acquisition request is detected, send and display the personality test questionnaire page on a user terminal pre-bound by the user; when it is detected that the data is filled in, generate a personality test result based on the personality test data on the personality test questionnaire page; when a test result sending instruction is received, encrypt and transmit the personality test result, and convert the received personality test result into a personalized feature according to the personalized data input format.

[0092] As an exemplary implementation of the above embodiment, the above multimodal feature collection module 701 can also be further used to: pre-construct a personality test questionnaire table, and set the display trigger condition of the personality test questionnaire page to a personalized data collection request; wherein the personality test questionnaire table includes a first personality option, a second personality option, a third personality option, a fourth personality option and a fifth personality option; the first personality option, the second personality option, the third personality option, the fourth personality option and the fifth personality option all have personality scores, a first personality aspect, a second personality aspect and a third personality aspect; each first personality aspect, each second personality aspect and each third personality aspect has a corresponding personality aspect score; obtain personality test text data including Big Five personality test questions and scores of each question, and fill in the personality test questionnaire table based on the personality test text data to generate a personality test questionnaire page.

[0093] Exemplarily, in some other implementations of the present embodiment, the above-mentioned time series feature processing module 702 can also be used to: obtain the implicit state data of various types of time series features of the target type features at each time step; perform multi-head attention calculation on each current implicit state data of the current time step to obtain multi-head weighted vectors corresponding to various types of time series features; based on the multi-head weighted vectors corresponding to various types of time series features, perform dimensionality reduction processing on various types of time series features, normalize the concatenated vectors of various types of time series features after dimensionality reduction, and obtain mixed time series data of the current time step; obtain fused emotional response features based on the mixed time series features of each time step.

[0094] As an exemplary implementation method of the above embodiment, the above-mentioned timing feature processing module 702 can also be further used for: splicing each current implicit state data into an implicit state splicing vector, and performing multi-head replication of the implicit state splicing vector according to the number of heads; each head independently and in parallel calculates the attention score: performing multi-head weight pre-calculation on the implicit state splicing vector to obtain a weighted pre-vector; based on the feature dimension of each current implicit state data, the implicit state splicing vector is divided into a plurality of sub-implicit state splicing vectors corresponding to the number of heads; for each head based on the weighted pre-vector, the attention score of each sub-implicit state splicing vector is calculated respectively, and each attention score is spliced ​​to obtain a total attention score; the attention weighted outputs of each head are spliced ​​to obtain a weighted total vector, and the multiple weighted total vectors are divided according to the order of each type of timing features to obtain multi-head weighted vectors corresponding to each type of timing features.

[0095] From the perspective of functional modules, see Figure 8 , Figure 8 The structure diagram of the emotion recognition device provided in this embodiment under a specific implementation manner, the device may include: The model training module 801 is used to obtain an emotion recognition training sample set and build an emotion recognition model; the emotion recognition model includes an input layer, a multimodal feature processing sub-model and an emotion recognition sub-model, and the multimodal feature processing sub-model includes a feature extraction layer, an adaptive alignment module, a temporal hybrid attention module and a feature dimension reduction module; using the emotion recognition training sample set, the emotion recognition model is cyclically trained based on the target classification loss function until the preset model training stop condition is reached.

[0096] The task execution module 802 is used to input the user's emotional response data of the target object and the user's personality test questionnaire data into the trained emotion recognition model. The input layer is used to receive the user's emotional response data of the target object and the user's personality test questionnaire data; the feature extraction layer extracts text feature data and target type features from the emotional response data, and generates personalized features according to the user's personality test questionnaire data; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; the adaptive alignment module aligns various types of time series features of the target type features in the time dimension; the time series mixed attention module fuses various types of time series features at each time step based on the attention mechanism to obtain a fused emotional response feature; the feature dimensionality reduction module performs dimensionality reduction processing on the fused emotional response features, text feature data and personalized features; the emotion recognition sub-model splices the emotional response compression features, text compression features and personality compression features into multimodal feature data, performs emotion recognition on the multimodal feature data, and outputs the emotion recognition result.

[0097] Exemplarily, in some other implementations of the present embodiment, the feature extraction layer includes one or more of an audio feature extraction model, a face recognition model, a visual feature extraction model, and a language feature extraction model; accordingly, the task execution module 802 can also be used to: input the emotional response video data into the audio feature extraction model, and obtain audio timing features according to the output of the audio feature extraction model; input the emotional response video data into the face recognition model, perform face recognition and face feature extraction according to the face recognition model, and obtain face timing features; input the emotional response video data into the visual feature extraction model, and obtain video timing features according to the output of the visual feature extraction model; convert the audio data of the emotional response video data into text data, and input the text data into the language feature extraction model, and obtain text feature data according to the output of the language feature extraction model.

[0098] Exemplarily, in some other implementations of this embodiment, the feature dimensionality reduction module includes a text information compression model; the text information compression model includes a first fully connected layer and a second fully connected layer; the number of neurons in the first fully connected layer matches the feature dimension of the text feature data, and the first fully connected layer has an activation function; accordingly, the task execution module 802 can also be used to: input the text feature data into the text information compression model, and obtain the text compression features corresponding to the text feature data according to the output of the text information compression model.

[0099] Exemplarily, in some other implementations of this embodiment, the feature dimensionality reduction module includes a personalized information compression model; the personalized information compression model includes a third fully connected layer and a fourth fully connected layer; the number of neurons in the third fully connected layer matches the feature dimension of the personalized feature, and the third fully connected layer has an activation function; accordingly, the task execution module 802 can also be used to: input the personalized feature into the personalized information compression model, and obtain the personalized compression feature corresponding to the personalized feature according to the output of the personalized information compression model.

[0100] Exemplarily, in some other implementations of the present embodiment, the adaptive alignment module includes multiple sub-time dimension alignment models, and the number of sub-time dimension alignment models is the same as the number of time series feature types; each sub-time dimension alignment model includes a first vector dimension rearrangement layer, an adaptive maximum pooling layer and a second vector dimension rearrangement layer.

[0101] Exemplarily, in some other implementations of this embodiment, the adaptive maximum pooling layer has a time step hyperparameter, and accordingly, the model training module 801 can also be used for: in the process of training the emotion recognition model, the time step hyperparameter is continuously adjusted through the back propagation algorithm, and the target time step hyperparameter that meets the preset model verification performance is used as the time step hyperparameter for performing the emotion recognition task.

[0102] Exemplarily, in some implementations of the present embodiment, the time series hybrid attention module includes a time series data modeling layer, a multi-head attention calculation module and a hybrid time series output module; wherein the time series data modeling layer includes a recurrent neural network for time series data modeling; the multi-head attention calculation module includes an implicit vector splicing layer, a fifth fully connected layer, a softmax (normalized exponential function layer), a score splicing layer and a multi-head output splicing layer; the hybrid time series output module includes a vector partitioning layer, a dimensionality reduction layer, a dimensionality reduction vector splicing layer and a vector normalization layer.

[0103] As an exemplary implementation method of the above embodiment, the model parameters of the multi-head attention calculation module include the number of heads. Accordingly, the model training module 801 can also be used for: in the process of training the emotion recognition model, the number of heads of the emotion recognition model is continuously adjusted through the back propagation algorithm, and the target number of heads that meets the preset model verification performance is used as the number of heads used to perform the emotion recognition task.

[0104] As another exemplary implementation of the above embodiment, the time series data modeling layer is a time series feature mapping model constructed based on a long short-term memory neural network, and the time series feature mapping model includes multiple sub-time series feature mapping models, and the number of each sub-time series feature mapping model is the same as the number of time series feature types; accordingly, the task execution module 802 can also be used to: for each type of time series feature, the mixed time series data of the previous time step, the current time series feature and its corresponding implicit state data and cell state data in the previous time step are respectively input into the corresponding current sub-time series feature mapping model, and according to the output of the current sub-time series feature mapping model, the implicit state data of the current time series feature in the current time step is obtained.

[0105] As an exemplary implementation method of the above embodiment, the feature dimension reduction module includes a time series mixed information compression model; the time series mixed information compression model includes a vector splicing layer, a sixth fully connected layer and a seventh fully connected layer; the number of neurons in the sixth fully connected layer matches the feature dimension of the vector splicing layer, and the sixth fully connected layer has an activation function; accordingly, the task execution module 802 can also be used to: input the mixed time series data of each time step and the implicit state data of various time series features into the time series mixed information compression model, the vector splicing layer splices the mixed time series data of the same time step and the implicit state data of various time series features, and obtains the emotional response compression feature corresponding to the fused emotional response feature through the sixth fully connected layer and the seventh fully connected layer.

[0106] The emotion recognition device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Fig. 9 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Fig. 9 As shown, the electronic device includes a memory 90 for storing computer programs; a processor 91 for implementing the steps of the emotion recognition method mentioned in any of the above embodiments when executing the computer program.

[0107] Among them, the processor 91 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 91 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 91 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 91 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 91 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 91 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0108] The memory 90 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 90 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the memory 90 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 90 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 90 may also include both an internal storage unit of an electronic device and an external storage device. The memory 90 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the emotion recognition method, etc., but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 90 is at least used to store the following computer program 901, wherein, after the computer program is loaded and executed by the processor 91, the relevant steps of the emotion recognition method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 90 may also include an operating system 902 and data 903, and the storage method may be temporary storage or permanent storage. The operating system 902 may include Windows, Unix, Linux, etc. The data 903 may include but is not limited to data corresponding to the emotion recognition result.

[0109] In some embodiments, the electronic device may further include a display screen 92, an input / output interface 93, a communication interface 94 or a network interface, a power supply 95 and a communication bus 96. Among them, the display screen 92 and the input / output interface 93, such as a keyboard, belong to the user interface, and exemplary user interfaces may also include standard wired interfaces, wireless interfaces, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 94 may exemplarily include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which are generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 96 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the figure, for example, it may also include a sensor 97 for realizing various functions.

[0111] It is understandable that if the emotion recognition method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the emotion recognition method recorded in any of the above embodiments are executed.

[0112] It is understandable that if the emotion recognition method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to perform all or part of the steps of the methods of each embodiment of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product storing a computer program, which, when executed by a processor, performs the steps of the emotion recognition method described in any of the above embodiments.

[0113] Finally, the present invention also provides an emotion recognition system, see Fig.10The emotion recognition system may include a first computing device 101 and a second computing device 102. The first computing device 101 and the second computing device 102 are connected via a target bus, such as a PCIE (peripheral component interconnect express, high-speed serial computer expansion bus) connection, such as a PCIe Gen5×16 of the second computing device 102 connected to the first computing device 101. The second computing device 102 has parallel computing capabilities. The first computing device 101 may be, for example, a device with a CPU as a processor, such as a server or a host computer. The second computing device 102 may be, for example, an FPGA or a GPU. In combination with the training process and the inference process of the emotion recognition model of the present invention, the second computing device 102 may be an FPGA.

[0114] Among them, the first computing device 101 includes a human-computer interaction module and an emotional response data collection module. When an emotion recognition task is received, the target object is displayed through the human-computer interaction module, and the emotional response data collection module is triggered to collect the emotional response data of the user perceiving the target object; a personality test questionnaire page is sent to the user terminal pre-bound to the user, and the personality test questionnaire data sent by the user terminal is received; the first computing device 101 sends the emotional response data and the personality test questionnaire data to the second computing device 102 through the target bus interface; the second computing device 102 implements the steps of the emotion recognition method recorded in any of the above embodiments when executing the computer program stored in the memory.

[0115] In order to further improve the data processing capability of the second computing device 102, based on the above embodiment, the present invention further provides an exemplary structure of the second computing device 102, which may include the following contents: In this embodiment, if Fig.11As shown, the second computing device 102 may include a data storage device, a data distribution module, a text information processing module, a personal information processing module, a tri-modal timing processing module, and an emotion recognition module. Among them, the text information processing module, the personal information processing module, the tri-modal timing processing module, and the emotion recognition module include both software and hardware programs, which are system modules supported by hardware by combining computer programs that implement corresponding functions with the hardware of the second computing device. The data distribution module, the text information processing module, the personal information processing module, the tri-modal timing processing module, and the emotion recognition module adopt an asynchronous communication method; in order to further improve the data communication efficiency inside the second computing device 102 and between the second computing device 102 and the first computing device, the functional modules inside the second computing device 102 can be interconnected through a NoC with high bandwidth, and the NoC can also be connected to Ethernet (Ethernet 400GE). On this basis, the RoCE-Lite (protocol name) protocol can also be carried through Ethernet, so that remote storage access can be performed through high-speed Ethernet to support the calculation of massive data.

[0116] Among them, the data distribution module is configured to extract text data and time series data from the emotional response data, send the text data to the text information processing module, input the personality test questionnaire data into the personal information processing module, and input the time series data into the trimodal time series processing module; the text information processing module is configured to extract text feature data from the text data, perform dimensionality reduction processing on the text feature data, and input the text compression feature into the emotion recognition module; the personal information processing module is configured to generate personalized features according to the personality test questionnaire data, perform dimensionality reduction processing on the personalized features, and input the personalized compression feature into the emotion recognition module; the trimodal time series processing module is configured to extract the target from the time series data The target type features are aligned in the time dimension with various time series features of the target type features, and various time series features at each time step are fused based on the attention mechanism to obtain the fused emotional response features, the fused emotional response features are reduced in dimension, and the emotional response compression features are input into the emotion recognition module; the target type features include at least two types of time series features among the audio time series features, the face time series features, and the video time series features; the emotion recognition module is configured to splice the emotional response compression features, the text compression features, and the personality compression features into multimodal feature data, perform emotion recognition on the multimodal feature data, and display the emotion recognition results to the user through the target bus interface and the human-computer interaction module.

[0117] In this embodiment, considering the differences in the data processing flow of feature data of different modes, among which the audio timing features, face timing features and video timing features are strongly coupled, the data processing flow of these three parts is processed by the above-mentioned tri-modal timing processing module. Due to the existence of the attention mechanism, although parallel computing is used in the multi-head attention process, the overall computational complexity and structural complexity are far greater than the text information processing module and the personal information processing module. Therefore, in hardware deployment, the text data, personalized data and target type data with strong coupling are distributed to three different modules through the data distribution module: the text information processing module, the personal information processing module and the tri-modal timing processing module. For the part of the data input to the board by the bus that cannot be processed in time, it can be temporarily stored in the data storage device. The data storage device can use DDR (Double Data Rate, double rate storage) for example, to store the high-speed access data required in the process of the association recognition model processing. In order to improve the data reading efficiency, the read and write operations of DDR are controlled by DDR Ctrl (Double Data Rate Controller, memory controller). In order to specialize different functional modules and reduce the power consumption of the entire system, the modules on the entire board do not use a global clock, but an asynchronous communication method. Work only occurs in the modules that need to work, and asynchronous communication can be achieved through the asynchronous alignment modules of each module. In order to achieve a fast output response, the output of different processing modules will be checked in the asynchronous alignment module of each module. If all modal information of a sample is processed, an event operation will be triggered, output to the emotion recognition module, and the output result will be returned to the first computing device 101. Exemplarily, considering that the text information processing module is simple to process, the data processing efficiency of this module is the fastest. After processing the data, it will send a request to send data signal to the asynchronous alignment module of each module. The asynchronous alignment module of each module will return a confirmation signal to the text information processing module, and then send the data to the asynchronous alignment module; in contrast, the processing speed of the tri-modal timing processing module is slow. After the processing of this module is completed, it will also be sent to the asynchronous alignment module of each module; the asynchronous alignment module of each module does not need to synchronize the clock with other modules, but as long as the internal confirmation that all the required processing is achieved, it will be directly input to the emotion recognition module for recognition. In this way, the processing of each module does not need to be strictly carried out step by step according to a global clock, but can flexibly decide the next operation based on the data processing speed of the module itself.

[0118] From the above, it can be seen that this embodiment reduces the data movement in the space of the second computing device such as the FPGA board by distributing different types of features to different system modules for processing; each module adopts a heterogeneous communication method to enhance the response of the emotion recognition model, effectively improve the data processing efficiency of the emotion recognition model, and then improve the execution efficiency of the emotion recognition task, and efficiently output the emotion recognition results.

[0119] Based on the above emotion recognition system, the second computing device is FPGA, the first computing device is the host computer, and the emotion recognition model is as follows: Figure 3 As shown, the present invention also provides a training, reasoning and hardware deployment implementation method of the emotion recognition model, which may include the following contents: A1: Divide the emotion recognition sample set into an emotion recognition training sample set, an emotion recognition verification set and an emotion recognition test set according to a preset ratio, such as 8:1:1 or 7:2:1.

[0120] A2: Extract faces from the emotion response videos in the emotion recognition training sample set, use the pre-trained face detection model to extract the corresponding face timing features, audio timing features, video timing features, and text features, and normalize the standard values ​​corresponding to the personality test questionnaire data in the emotion recognition training sample set to obtain personalized features.

[0121] A3: Input the personalized features into the personalized information compression model to obtain the corresponding personalized compression features.

[0122] A4: Input the text feature data into the text information compression model to obtain the corresponding text compression features.

[0123] A5: Input the audio timing features, face timing features and video timing features into the adaptive alignment module for length alignment.

[0124] A6: Input the output of A5 into the temporal hybrid attention module to obtain four output vectors.

[0125] A7: Concatenate the output of A6 into a vector and input it into the temporal mixed information compression model to obtain the emotional response compression feature.

[0126] A8: Concatenate the outputs of A3, A4, and A7, and input the concatenated features into the emotion recognition module to obtain the output of emotion recognition.

[0127] A9: Use the cross entropy loss function to measure the distance between the prediction results of A8 and the true label, and use the back propagation algorithm to train the weights of the emotion recognition model on the emotion recognition training sample set.

[0128] A10: Verify on the emotion recognition verification set to select the emotion recognition model weights with the best performance and fix them for hardware deployment.

[0129] A11: Based on Fig.11 The host computer + FPGA on-board structure shown performs hardware deployment of the emotion recognition model and quickly returns the emotion recognition results through asynchronous alignment.

[0130] A12: When using the emotion recognition system of A11 to perform emotion recognition tasks, personalized information containing sensitive information can be saved locally, such as being input to the FPGA board through a USB flash drive peripheral.

[0131] As can be seen from the above, this embodiment forms an emotion recognition system through a host computer + FPGA, which can not only effectively improve the training effect of the emotion recognition model, but also facilitate the efficient output of emotion recognition results. In addition, the emotion recognition model can learn more accurate emotion recognition capabilities from fully considering personalized factors and integrating audio-visual, text and personalized information, effectively improving the execution accuracy of the emotion recognition task. Furthermore, the features of different time dimensions are fused after alignment, and the features of different dimensions complement each other, which is conducive to improving the accuracy and richness of the input feature representation of the emotion recognition model, and further improving the execution accuracy of the emotion recognition task. By reducing the dimensionality of features of different dimensions, the data processing efficiency of the emotion recognition model can be accelerated, and efficient and high-accuracy emotion recognition can be achieved.

[0132] The above is a detailed introduction to an emotion recognition method, device, electronic device, non-volatile storage medium and computer program product provided by the present invention. The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can refer to each other. Whether the units and algorithm steps of each example described in each disclosed embodiment are executed in electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods for each specific application to implement the described functions, and such implementation should not be considered to exceed the scope of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. An emotion recognition method, characterized in that: include: Extract text feature data and target type features from the emotional response data of the user's perception of the target object; The target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; Generate personalized features based on the user's personality test questionnaire data; Aligning various time series features of the target type features in the time dimension, and fusing various time series features at each time step based on the attention mechanism to obtain a fused emotional response feature; The fused emotional response feature, the text feature data and the personalized feature are subjected to dimensionality reduction processing to obtain multimodal feature data for emotion recognition.

2. The emotion recognition method according to claim 1, characterized in that: Align various time series features of the target type features in the time dimension, including: Rearranging the feature dimensions of various time series features of the target type feature to replace the second dimension and the third dimension of the feature vector corresponding to each time series feature; For various time series features after dimension permutation, adaptive maximum pooling operation is performed on the new third dimension to output various time series features with the same third dimension; According to the dimension output format of the target type feature, the feature dimensions of each type of time series features are rearranged again.

3. The emotion recognition method according to claim 2, characterized in that: Before aligning various time series features of the target type features in the time dimension, it also includes: When receiving a network size adjustment instruction, parsing the network size adjustment instruction to obtain a time step hyperparameter; Based on the time step hyperparameter, an adaptive maximum pooling operation is performed on the various time series features after the dimension is permuted to obtain various time series features whose third dimensions are all the time step hyperparameters.

4. The emotion recognition method according to claim 1, characterized in that: Generate personalized features based on the user's personality test questionnaire data, including: When a personalized data collection request is detected, the personality test questionnaire page is sent and displayed on the user's pre-bound user terminal; When it is detected that the data is completed, a personality test result is generated according to the personality test data on the personality test questionnaire page; When a test result sending instruction is received, the personality test result is encrypted and transmitted, and the received personality test result is converted into personalized features according to the personalized data input format.

5. The emotion recognition method according to claim 4, characterized in that: Before sending the personality test questionnaire page to the user's pre-bound client and displaying it, it also includes: Pre-constructing a personality test questionnaire, and setting a display trigger condition of the personality test questionnaire page as a personalized data collection request; The personality test questionnaire includes a first personality option, a second personality option, a third personality option, a fourth personality option and a fifth personality option; the first personality option, the second personality option, the third personality option, the fourth personality option and the fifth personality option all have personality scores, first personality aspects, second personality aspects and third personality aspects; each first personality aspect, each second personality aspect and each third personality aspect has a corresponding personality aspect score; Obtain personality test text data including Big Five personality test questions and scores for each question, and fill in the personality test questionnaire table based on the personality test text data to generate a personality test questionnaire page.

6. The emotion recognition method according to any one of claims 1 to 5, characterized in that: Based on the attention mechanism, various temporal features at each time step are fused to obtain fused emotional response features, including: Obtaining implicit state data of various time series features of the target type features at each time step; Perform multi-head attention calculation on each current implicit state data of the current time step to obtain multi-head weighted vectors corresponding to various time series features; based on the multi-head weighted vectors corresponding to various time series features, perform dimensionality reduction processing on various time series features, and normalize the concatenated vectors of various time series features after dimensionality reduction to obtain mixed time series data of the current time step; The fused emotional response features are obtained based on the mixed temporal features of each time step.

7. The emotion recognition method according to claim 6, characterized in that: Perform multi-head attention calculation on each current implicit state data of the current time step, including: splicing each current implicit state data into an implicit state splicing vector, and performing multi-head replication on the implicit state splicing vector according to the number of heads; Each head calculates the attention score independently and in parallel: pre-calculate the multi-head weights of the implicit state splicing vector to obtain a weighted pre-vector; based on the feature dimensions of each current implicit state data, divide the implicit state splicing vector into a plurality of sub-implicit state splicing vectors corresponding to the number of heads; based on the weighted pre-vector, calculate the attention score of each sub-implicit state splicing vector for each head, and splice the attention scores to obtain the total attention score; The weighted outputs of the attention of each head are concatenated to obtain a weighted total vector, and the multiple weighted total vectors are divided according to the order of various types of temporal features to obtain multi-head weighted vectors corresponding to various types of temporal features.

8. An emotion recognition method, characterized in that: include: Obtain emotion recognition training sample sets and build emotion recognition models; The emotion recognition model includes an input layer, a multimodal feature processing submodel and an emotion recognition submodel, and the multimodal feature processing submodel includes a feature extraction layer, an adaptive alignment module, a temporal hybrid attention module and a feature dimension reduction module; Using the emotion recognition training sample set, the emotion recognition model is cyclically trained based on the target classification loss function until a preset model training stop condition is reached; The emotional response data of the user perceiving the target object and the personality test questionnaire data of the user are input into the trained emotion recognition model, wherein the input layer is used to receive the emotional response data of the user perceiving the target object and the personality test questionnaire data of the user; the feature extraction layer extracts text feature data and target type features from the emotional response data, and generates personalized features according to the personality test questionnaire data of the user; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; The adaptive alignment module aligns various time series features of the target type features in the time dimension; the time series hybrid attention module fuses various time series features at each time step based on the attention mechanism to obtain a fused emotional response feature; The feature dimension reduction module performs dimension reduction processing on the fused emotional response features, the text feature data and the personalized features; the emotion recognition sub-model splices the emotional response compression features, text compression features and personality compression features into multimodal feature data, performs emotion recognition on the multimodal feature data, and outputs the emotion recognition results.

9. The emotion recognition method according to claim 8, characterized in that: The emotional response data is emotional response video data, and the feature extraction layer includes one or more of an audio feature extraction model, a face recognition model, a visual feature extraction model, and a language feature extraction model; extracting text feature data and target type features from the emotional response data includes: Inputting the emotional response video data into the audio feature extraction model, and obtaining audio time series features according to the output of the audio feature extraction model; Inputting the emotional response video data into the face recognition model, performing face recognition and face feature extraction according to the face recognition model, and obtaining face temporal features; Inputting the emotional response video data into the visual feature extraction model, and obtaining video temporal features according to the output of the visual feature extraction model; The audio data of the emotion response video data is converted into text data, and the text data is input into the language feature extraction model, and the text feature data is obtained according to the output of the language feature extraction model.

10. The emotion recognition method according to claim 8, characterized in that: The feature dimension reduction module includes a text information compression model; the text information compression model includes a first fully connected layer and a second fully connected layer; the number of neurons in the first fully connected layer matches the feature dimension of the text feature data, and the first fully connected layer has an activation function; The dimension reduction process of the text feature data includes: inputting the text feature data into the text information compression model, and obtaining text compression features corresponding to the text feature data according to the output of the text information compression model.

11. The emotion recognition method according to claim 8, characterized in that: The feature dimension reduction module includes a personalized information compression model; the personalized information compression model includes a third fully connected layer and a fourth fully connected layer; the number of neurons in the third fully connected layer matches the feature dimension of the personalized feature, and the third fully connected layer has an activation function; The dimension reduction process of the personalized feature includes: inputting the personalized feature into the personalized information compression model, and obtaining the personalized compression feature corresponding to the personalized feature according to the output of the personalized information compression model.

12. The emotion recognition method according to claim 8, characterized in that: The adaptive alignment module includes multiple sub-time dimension alignment models, and the number of each sub-time dimension alignment model is the same as the number of time series feature types; each sub-time dimension alignment model includes a first vector dimension rearrangement layer, an adaptive maximum pooling layer and a second vector dimension rearrangement layer.

13. The emotion recognition method according to claim 12, characterized in that: The adaptive maximum pooling layer has a time step hyperparameter. During the training process of the emotion recognition model, the time step hyperparameter is continuously adjusted through the back propagation algorithm, and the target time step hyperparameter that meets the preset model verification performance is used as the time step hyperparameter for performing the emotion recognition task.

14. The emotion recognition method according to claim 8, characterized in that: The temporal hybrid attention module includes a temporal data modeling layer, a multi-head attention calculation module and a hybrid temporal output module; Among them, the time series data modeling layer includes a recurrent neural network for time series data modeling; the multi-head attention calculation module includes an implicit vector splicing layer, a fifth fully connected layer, a normalized exponential function layer, a score splicing layer and a multi-head output splicing layer; the mixed time series output module includes a vector partitioning layer, a dimensionality reduction layer, a dimensionality reduction vector splicing layer and a vector normalization layer.

15. The emotion recognition method according to claim 14, characterized in that: The model parameters of the multi-head attention calculation module include the number of heads. During the training process of the emotion recognition model, the number of heads of the emotion recognition model is continuously adjusted through the back propagation algorithm, and the target number of heads that meets the preset model verification performance is used as the number of heads used to perform the emotion recognition task.

16. The emotion recognition method according to claim 14, characterized in that: The time series data modeling layer is a time series feature mapping model constructed based on a long short-term memory neural network, and the time series feature mapping model includes a plurality of sub-time series feature mapping models, and the number of each sub-time series feature mapping model is the same as the number of time series feature types; The generation process of the implicit state data of various types of time series features of the target type features at each time step includes: for each type of time series features, the mixed time series data of the previous time step, the current time series features and their corresponding implicit state data and cell state data at the previous time step are respectively input into the corresponding current sub-time series feature mapping model, and the implicit state data of the current time series feature at the current time step is obtained according to the output of the current sub-time series feature mapping model.

17. The emotion recognition method according to claim 16, characterized in that: The feature dimension reduction module includes a temporal mixed information compression model; the temporal mixed information compression model includes a vector splicing layer, a sixth fully connected layer and a seventh fully connected layer; the number of neurons in the sixth fully connected layer matches the feature dimension of the vector splicing layer, and the sixth fully connected layer has an activation function; The dimensionality reduction process of the fused emotional response feature includes: inputting the mixed time series data of each time step and the implicit state data of each type of time series feature into the time series mixed information compression model, the vector splicing layer splicing the mixed time series data of the same time step and the implicit state data of each type of time series feature, and passing the time series mixed splicing vector of the current time step through the sixth fully connected layer and the seventh fully connected layer to obtain the emotional response compression feature corresponding to the fused emotional response feature.

18. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to implement the steps of the emotion recognition method as claimed in any one of claims 1 to 17 when executing the computer program stored in the memory.

19. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the emotion recognition method according to any one of claims 1 to 17 are implemented.

20. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the emotion recognition method described in any one of claims 1 to 17 are implemented.

21. An emotion recognition system, characterized in that: The device comprises a first computing device and a second computing device, wherein the first computing device and the second computing device are connected via a target bus, and the second computing device has a parallel computing capability; The first computing device includes a human-computer interaction module and an emotional response data collection module. When receiving an emotion recognition task, the target object is displayed through the human-computer interaction module, and the emotional response data collection module is triggered to collect the emotional response data of the user perceiving the target object; a personality test questionnaire page is sent to a user terminal pre-bound by the user, and the personality test questionnaire data sent by the user terminal is received; The first computing device sends the emotional response data and the personality test questionnaire data to the second computing device via a target bus interface; The second computing device implements the steps of the emotion recognition method as described in any one of claims 1 to 17 when executing the computer program stored in the memory.

22. The emotion recognition system according to claim 21, characterized in that: The second computing device includes a data storage device, a data distribution module, a text information processing module, a personal information processing module, a tri-modal timing processing module and an emotion recognition module, wherein the data distribution module, the text information processing module, the personal information processing module, the tri-modal timing processing module and the emotion recognition module adopt an asynchronous communication mode; Wherein, the data distribution module is configured to extract text data and time series data from the emotional response data, send the text data to the text information processing module, input the personality test questionnaire data to the personal information processing module, and input the time series data to the tri-modal time series processing module; The text information processing module is configured to extract text feature data from the text data, perform dimensionality reduction processing on the text feature data, and input text compression features into the emotion recognition module; The personal information processing module is configured to generate personalized features according to the personality test questionnaire data, perform dimensionality reduction processing on the personalized features, and input the personality compression features into the emotion recognition module; The trimodal time series processing module is configured to extract target type features from time series data, align various time series features of the target type features in the time dimension, fuse various time series features at each time step based on the attention mechanism, obtain fused emotional response features, perform dimensionality reduction processing on the fused emotional response features, and input the emotional response compression features into the emotion recognition module; the target type features include at least two types of time series features among audio time series features, face time series features, and video time series features; The emotion recognition module is configured to splice the emotion response compression feature, the text compression feature and the personality compression feature into multimodal feature data, perform emotion recognition on the multimodal feature data, and display the emotion recognition result to the user through the target bus interface and the human-computer interaction module.

Images