Emotion recognition method, device, terminal device and computer-readable storage medium

Abstract

The present application discloses an emotion recognition method, device, terminal equipment and computer-readable storage medium, the method comprising: extracting the face area of ​​each frame image as a training sample based on an input continuous frame video sequence; inputting the training sample into a three-dimensional convolutional neural network model to obtain a global feature map of each video frame sequence; dividing the global feature map into blocks, using the generated local feature map and the global feature map as feature blocks, extracting one-dimensional features of each feature block and inputting them into the three-dimensional convolutional neural network model, outputting an initial predicted emotion score, and determining an initial weight of each feature block, determining a corresponding target weight according to the initial weight of each feature block; using the target weight to weight the initial predicted emotion score to generate a target predicted emotion score. The present application combines a three-dimensional convolutional neural network and feature map blocks to enhance the confidence of the unobstructed part, improve the accuracy and efficiency of emotion recognition, and has a wider range of applications.

Description

Technical Field

[0001] The present application relates to the technical field of emotion recognition, and in particular to an emotion recognition method, apparatus, terminal device and computer-readable storage medium. Background Art

[0002] Emotion recognition is a research hotspot in the field of artificial intelligence and has been widely used in the fields of robot manufacturing, human-computer interaction, communication, autonomous driving, and health monitoring. In the field of health monitoring, the health monitoring system captures the health emotions of workers in real time by analyzing the facial expressions of workers in live videos, which is conducive to timely detection and guidance of mental health problems. In practical applications, two emotion classification indicators are usually used for emotion recognition. One is based on discrete dimensions, in which the emotional classification is often relatively simple and easy to understand; the other is based on continuous dimensions, which is more in line with the characteristics of human emotional continuity.

[0003] At present, there are two common emotion recognition methods: one is to extract image frames from video streams and extract facial areas, perform expression recognition on the target, and combine the results of previous and next frames for weighted emotion recognition. This emotion recognition method for video streams not only does not take into account the possible occlusion in the actual scene, but also needs to first recognize a single frame image during calculation, and then make a comprehensive evaluation of the results of multiple frames. Therefore, the operation efficiency is low, and the ability to capture the temporal correlation of the emotions of people in the video stream is poor. The other is to use a deep convolutional neural network with a multi-way self-attention mechanism to obtain the local features, occlusion features, and overall features of the student's expression image, and obtain the expression features by adaptively weighting and merging these features to recognize the student's expression. Although this method takes into account the facial occlusion, it can only process single-frame images and belongs to static emotion recognition. Therefore, this method may not accurately reflect the real emotional situation of people. Summary of the invention

[0004] The purpose of the present application is to provide an emotion recognition method, apparatus, terminal device and computer-readable storage medium to solve the problems of poor operating efficiency and low recognition accuracy in existing emotion recognition methods.

[0005] To achieve the above objectives, the present application provides an emotion recognition method, comprising:

[0006] Based on the input continuous frame video sequence, the face area of ​​each frame image is extracted as a training sample;

[0007] Input the training samples into the 3D convolutional neural network model to obtain the global feature map of each video frame sequence;

[0008] The global feature map is divided into blocks, the generated local feature map and the global feature map are used as feature blocks, and the one-dimensional features of each feature block are extracted;

[0009] Input the one-dimensional features into the three-dimensional convolutional neural network model, output the initial predicted emotion score, and determine the initial weight of each feature block, and determine the corresponding target weight according to the initial weight of each feature block;

[0010] The initial predicted sentiment score is weighted by the target weight to generate a target predicted sentiment score.

[0011] Further, preferably, before extracting the face area of ​​each frame image as a training sample, the method includes:

[0012] Extract the face area of ​​each frame image and get the corresponding rectangular frame coordinates and confidence:

[0013] B i ,PR i =YOLO(frame t );

[0014] In the formula, frame t represents the tth video frame, B i Represents the coordinates of the upper left corner and lower right corner of the rectangular box marking the i-th face in the video frame, PR i Represents the confidence of the i-th face;

[0015] Get the rectangular frame area with the highest confidence in each frame image, and use bilinear interpolation to process all rectangular frame areas to the same size.

[0016] Further, preferably, extracting the face region of each frame of image includes:

[0017] The YOLOv3 model pre-trained with the WIDER Face dataset is used to extract the face area of ​​each frame.

[0018] Further, preferably, the extracting one-dimensional features of each feature block includes:

[0019] A one-dimensional convolutional neural network model is used to extract one-dimensional features of each feature block.

[0020] Further, preferably, the determining the initial weight of each feature block includes:

[0021] Determine the loss function for each feature block and calculate the total loss function:

[0022] The total loss function threshold is set, and the 3D convolutional neural network model is trained using training samples until the model converges, and the initial weight of each feature block is determined at this time.

[0023] Further, preferably, the initial weight of each feature block is:

[0024]

[0025] In the formula, l i is the loss function of the ith feature block, w 1i is the weight of the i-th feature block.

[0026] Further, preferably, determining the corresponding target weight according to the initial weight of each feature block includes:

[0027] Based on the one-dimensional features of each feature block, the k-means algorithm is used to determine the data distribution center of the one-dimensional features of each feature block;

[0028] Define the Euclidean distance between each feature block and the data distribution center as weight w 2i ;

[0029] Determine the target weight of each feature block as w i =w 1i × 2i .

[0030] The present application also provides an emotion recognition device, comprising:

[0031] A training sample acquisition unit, used for extracting a face region of each frame image as a training sample based on an input continuous frame video sequence;

[0032] A global feature map acquisition unit, used for inputting training samples into a three-dimensional convolutional neural network model to obtain a global feature map of each video frame sequence;

[0033] A feature map block unit is used to block the global feature map, use the generated local feature map and the global feature map as feature blocks, and extract one-dimensional features of each feature block;

[0034] A target weight calculation unit is used to input the one-dimensional feature into the three-dimensional convolutional neural network model, output the initial predicted emotion score, and determine the initial weight of each feature block, and determine the corresponding target weight according to the initial weight of each feature block;

[0035] The emotion recognition unit is used to weight the initial predicted emotion score using the target weight to generate a target predicted emotion score.

[0036] The present application also provides a terminal device, including:

[0037] one or more processors;

[0038] A memory, coupled to the processor, for storing one or more programs;

[0039] When the one or more programs are executed by the one or more processors, the one or more processors implement the emotion recognition method as described in any one of the above items.

[0040] The present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the emotion recognition method as described in any one of the above items.

[0041] Compared with the prior art, the beneficial effects of this application are:

[0042] The present application discloses an emotion recognition method, apparatus, terminal device and computer-readable storage medium, the method comprising: extracting the face area of ​​each frame image as a training sample based on an input continuous frame video sequence; inputting the training sample into a three-dimensional convolutional neural network model to obtain a global feature map of each video frame sequence; dividing the global feature map into blocks, using the generated local feature map and the global feature map as feature blocks, and extracting one-dimensional features of each feature block; inputting the one-dimensional features into the three-dimensional convolutional neural network model, outputting an initial predicted emotion score, and determining an initial weight of each feature block, and determining a corresponding target weight according to the initial weight of each feature block; and weighting the initial predicted emotion score using the target weight to generate a target predicted emotion score.

[0043] This application directly extracts the temporal and spatial features of facial emotion information in video sequences through a three-dimensional convolutional neural network model, thereby improving the accuracy and efficiency of facial emotion recognition; by dividing the feature map of the video sequence into blocks and calculating an adaptive weight for each block, the confidence of the unobstructed part is enhanced and the confidence of the obstructed part is reduced, thereby solving the occlusion phenomenon that may exist in actual scenes and having a wider range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is a flowchart of an emotion recognition method provided by a certain embodiment of the present application;

[0046] Figure 2is a schematic diagram of the structure of an emotion recognition device provided by a certain embodiment of the present application;

[0047] Figure 3 It is a structural diagram of a terminal device provided in a certain embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0050] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0051] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0052] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0053] It should be noted that currently, two types of emotion classification indicators are commonly used in the field of emotion recognition: discrete dimension and continuous dimension. For discrete dimension, emotions are usually divided into 8 basic categories: anger, fear, sadness, hate, expectation, surprise, approval, and happiness. This emotion classification method is relatively simple and easy to understand, and has been widely used in emotion recognition research. For continuous dimension, a two-dimensional model is used to quantify emotions, dividing emotions into two dimensions: valence and arousal. The score range of each dimension is 1-9. Since the continuous dimension is more in line with the characteristics of human emotion continuity, the emotion recognition research on the continuous dimension is more practical.

[0054] Furthermore, emotion recognition can be divided into static facial emotion recognition and dynamic facial emotion recognition. The former hopes to identify people's expressions through a single picture to achieve the purpose of recognizing people's emotions, while the latter hopes to perceive people's changing expressions in video / picture sequences to understand people's emotions. In actual use, relying solely on pictures does not necessarily accurately reflect people's emotional state. In some scenarios, it is necessary to combine the changes in people's expressions to truly understand people's emotions, so the research on video-based facial emotion recognition is becoming more and more extensive. However, the existing emotion recognition method for video streams not only does not consider the possible occlusion in the actual scene, but also needs to recognize a single frame image first during calculation, and then comprehensively judge the results of multiple frames, so the operating efficiency is low, and the ability to capture the temporal correlation of people's emotions in the video stream is poor. Therefore, this embodiment aims to provide a dynamic emotion recognition that can be applied to facial occlusion situations, while improving recognition efficiency and recognition accuracy.

[0055] See also Figure 1 , a certain embodiment of the present application provides an emotion recognition method. Figure 1 As shown, the emotion recognition method includes steps S10 to S50. The specific steps are as follows:

[0056] S10. Based on the input continuous frame video sequence, extract the face area of ​​each frame image as a training sample.

[0057] In this step, firstly, an input continuous frame video sequence is obtained based on the captured video information, and then a face image is extracted based on the continuous frame video sequence to construct a training sample.

[0058] In one embodiment, before extracting the face region of each frame image as a training sample, the process includes:

[0059] Extract the face area of ​​each frame image and get the corresponding rectangular frame coordinates and confidence:

[0060] B i ,PR i =YOLO(frame t );

[0061] In the formula, frame t represents the tth video frame, B i Represents the coordinates of the upper left corner and lower right corner of the rectangular box marking the i-th face in the video frame, PR i Represents the confidence of the i-th face;

[0062] Get the rectangular frame area with the highest confidence in each frame image, and use bilinear interpolation to process all rectangular frame areas to the same size.

[0063] Preferably, the face region of each frame image is extracted using a YOLOv3 model pre-trained with the WIDER Face dataset.

[0064] In this embodiment, the acquired original video data is mainly preprocessed to make the sample quality of the extracted face image higher. Specifically:

[0065] 1.1) For the input continuous T-frame video sequence, the YOLOv3 model pre-trained with the WIDER Face dataset is used to extract the face area of ​​each frame. It should be noted that the earliest public version of the WIDER Face dataset contains 32,203 images with face annotations, a total of 393,703 face data, and each face data is accompanied by more detailed information, including blur, expression, illumination, occlusion, pose, etc. In this embodiment, after the YOLOv3 model is trained with the WIDER Face dataset, the quality of the extracted face area can be guaranteed to be higher.

[0066] 1.2) Input an image into the pre-trained YOLOv3 model to obtain the coordinates and confidence of the rectangular box that may be the face area in the image. The formula is as follows:

[0067] B i ,PR i =YOLO(frame t );

[0068] In the formula, frame t represents the tth video frame, B i Represents the coordinates of the upper left corner and lower right corner of the rectangular box marking the i-th face in the video frame, PR i Represents the confidence of the i-th face; the higher the confidence, the greater the probability that the calibrated target is a face.

[0069] 1.3) For each image, select the rectangular box with the highest confidence and cut out the corresponding area from the image, recorded as where h t ,w t is the length and width of the face area, and c is the number of channels.

[0070] Since the sizes of face regions extracted from different image frames are not necessarily the same, in order to unify the format, in this embodiment, bilinear interpolation is preferably used to scale the T frame images to the same dimension. Bilinear interpolation is also called bilinear interpolation. Bilinear interpolation is a linear interpolation extension of an interpolation function with two variables, and its core idea is to perform linear interpolation in two directions respectively.

[0071] Specifically, during the calculation, we first set the target frame size to H×W, and obtain:

[0072]

[0073] Then calculate the position of the target pixel in the original image, where dstX and dstY represent the pixel points of the target frame, and srcX and srcY represent the pixel points in the corresponding source image. Because the sizes of the source image and the target image are generally different, the calculated srcX and srcY are generally floating point numbers.

[0074] Suppose the floating point coordinates obtained by the inverse transformation are (i+u,j+v); where i and j are the integer parts of srcX and srcY, and u and v are the decimal parts, which are floating point numbers in the interval [0,1). Therefore, the value of this pixel f(i+u,j+v) can be determined by the values ​​of the four surrounding pixels corresponding to the coordinates (i,,j), (i+1,j), (i,j+1), and (i+1,j+1) in the original image, that is:

[0075] f(i+u,j+v)=(1-u)(1-v)f(i,j)+(1-u)vf(i,j+1)+u(1-v)f(i+1,j)+uvf(i+1,j+1);

[0076] Where f(i,j) represents the pixel value at (i,j) in the original image, and so on. Treated to the same size This is the preprocessed video frame.

[0077] S20, input the training samples into the three-dimensional convolutional neural network model to obtain a global feature map of each video frame sequence.

[0078] In this embodiment, a three-layer three-dimensional convolutional neural network model, namely a 3D CNN architecture, is used to extract the sample feature map. The architecture includes three convolutional layers and two downsampling layers. The extraction formula of the sample feature map is as follows:

[0079] F whole =Conv3d(SubSample(Conv3d(SubSample(Conv3d(FACE)))));

[0080] In the formula, is the preprocessed video frame sequence, H, W are the height and width of each frame, c is the number of channels, T is the number of frames in the video sequence, Conv3d() represents a three-dimensional convolution operation, SubSample() represents downsampling, and the dimension of the extracted feature map is

[0081] S30, dividing the global feature map into blocks, using the generated local feature map and the global feature map as feature blocks, and extracting one-dimensional features of each feature block.

[0082] In this step, in order to weaken the influence of the occluded part on the classification result, according to the feature map F in step S20 whole The height H' and width W' of are divided into multiple segments, and the divided regions and the global feature map are used as feature blocks.

[0083] As a preferred embodiment, F whole The height H' and width W' of the whole There are five feature blocks in total, and the one-dimensional features of these five blocks are extracted respectively. In a specific implementation, a one-dimensional convolutional neural network model, i.e., a 1D CNN model, is used to extract the one-dimensional features of each feature block. whole The extracted one-dimensional feature is recorded as A 0 , the one-dimensional features of the four region blocks are divided into A 1 , A 2 , A 3 , A 4 .

[0084] S40, inputting the one-dimensional feature into the three-dimensional convolutional neural network model, outputting the initial predicted emotion score, and determining the initial weight of each feature block, and determining the corresponding target weight according to the initial weight of each feature block.

[0085] In this embodiment, the one-dimensional feature is input into the fully connected layer of the three-dimensional convolutional neural network model to obtain the predicted emotion score i=0,1,2,3,4.

[0086] Furthermore, the initial weight of each feature block is determined, including:

[0087] 4.1) Determine the loss function of each feature block and calculate the total loss function:

[0088] 4.2) Set the total loss function threshold, use the training samples to train the 3D convolutional neural network model until the model converges, and determine the initial weight of each feature block at this time.

[0089] In this embodiment, the mean square error loss function is preferably used to calculate the loss of each feature block. Suppose the loss calculated for the feature blocks of the five parts is L i , i = 0, 1, 2, 3, 4, corresponding to the overall features and four local features respectively, then the final total loss function is:

[0090] L=L 0 +λ(L 1 +L2 +L 3 +L 4 );

[0091] Among them, λ is a weight parameter that balances the overall features and local features, which can usually be set to 0.5. According to the calculation formula of the above loss function, the threshold of the loss function can be set in advance, and then the three-dimensional convolutional neural network model is trained using training samples until the model converges. That is, when the loss function of the model output predicting the emotion score reaches the threshold of the loss function, the model converges, and at this time, the adaptive weight of each feature block is calculated.

[0092] Furthermore, when the model converges, the network parameters of the model are fixed. First, the loss value after the five feature blocks converge is calculated, denoted as l i , i = 0, 1, 2, 3, 4. It is generally believed that the smaller the loss after convergence, the more useful the corresponding features of this part are for the classification results. Therefore, in this embodiment, the inverse of these losses are normalized to obtain the weight parameter w of the feature block of each part. 1i , i = 0, 1, 2, 3, 4, and use this weight parameter as the initial weight of each feature block, that is:

[0093]

[0094] In the formula, l i is the loss function of the ith feature block, w 1i is the weight of the i-th feature block.

[0095] In one embodiment, after calculating the initial weight of each feature block, it is necessary to determine the corresponding target weight, which specifically includes:

[0096] 4.3) Based on the one-dimensional features of each feature block, the k-means algorithm is used to determine the data distribution center of the one-dimensional features of each feature block;

[0097] 4.4) Define the Euclidean distance between each feature block and the data distribution center as weight w 2i ;

[0098] 4.5) Determine the target weight of each feature block as w i =w 1i × 2i .

[0099] S50: weighting the initial predicted emotion score using the target weight to generate a target predicted emotion score.

[0100] Finally, in this step, the predicted emotion scores Y of the five blocks obtained in step S40 are i ％, and the target weight of each feature block calculated in step S50, to obtain the target predicted emotion score, that is:

[0101]

[0102] In summary, the embodiment of the present invention directly learns and represents the facial emotion information in a video sequence by using 3D CNN, which has higher accuracy than the method of identifying emotions by a single image, and has higher efficiency than the method of extracting each frame of the video sequence for identification and integrating the information of multiple frames. At the same time, the present invention can handle the situation where the face is partially blocked in the video, divide the extracted facial feature map into blocks, and design a method for calculating the importance weight of each block, and use adaptive weights to give different confidences to each block to weaken the influence of the blocked part on the whole. Finally, according to the calculated weights, the prediction results of each part are weighted for emotion analysis. On the one hand, the information of the local block of the video frame can more accurately reflect the local differences of different videos; on the other hand, the block division and calculation of the adaptive weights can make the algorithm better predict the emotions of the operator by adaptively increasing the weight of the unblocked part and reducing the weight of the blocked part when actually monitoring the operator, even if there is occlusion.

[0103] See also Figure 2 In one embodiment of the present application, an emotion recognition device is also provided, comprising:

[0104] The training sample acquisition unit 01 is used to extract the face area of ​​each frame image as a training sample based on the input continuous frame video sequence;

[0105] The global feature map acquisition unit 02 is used to input the training sample into the three-dimensional convolutional neural network model to obtain the global feature map of each video frame sequence;

[0106] The feature map block unit 03 is used to block the global feature map, use the generated local feature map and the global feature map as feature blocks, and extract the one-dimensional features of each feature block;

[0107] The target weight calculation unit 04 is used to input the one-dimensional feature into the three-dimensional convolutional neural network model, output the initial predicted emotion score, and determine the initial weight of each feature block, and determine the corresponding target weight according to the initial weight of each feature block;

[0108] The emotion recognition unit 05 is used to weight the initial predicted emotion score using the target weight to generate a target predicted emotion score.

[0109] In one embodiment, the training sample acquisition unit 01 is further configured to, before extracting the face region of each frame image as a training sample,

[0110] Extract the face area of ​​each frame image and get the corresponding rectangular frame coordinates and confidence:

[0111] B i ,PR i =YOLO(frame t );

[0112] In the formula, frame t represents the tth video frame, B i Represents the coordinates of the upper left corner and lower right corner of the rectangular box marking the i-th face in the video frame, PR i Represents the confidence of the i-th face;

[0113] Get the rectangular frame area with the highest confidence in each frame image, and use bilinear interpolation to process all rectangular frame areas to the same size.

[0114] In one embodiment, the training sample acquisition unit 01 is further configured to extract the face region of each frame image using a YOLOv3 model pre-trained with the WIDER Face dataset.

[0115] In one embodiment, the feature map blocking unit 03 is also used to extract one-dimensional features of each feature block using a one-dimensional convolutional neural network model.

[0116] In one embodiment, the target weight calculation unit 04 is further used to:

[0117] Determine the initial weight of each feature block, including:

[0118] Determine the loss function for each feature block and calculate the total loss function:

[0119] The total loss function threshold is set, and the 3D convolutional neural network model is trained using training samples until the model converges, and the initial weight of each feature block is determined at this time.

[0120] Among them, the initial weight of each feature block is:

[0121]

[0122] In the formula, l i is the loss function of the ith feature block, w 1i is the weight of the i-th feature block.

[0123] Determine the corresponding target weight according to the initial weight of each feature block, including:

[0124] Based on the one-dimensional features of each feature block, the k-means algorithm is used to determine the data distribution center of the one-dimensional features of each feature block;

[0125] Define the Euclidean distance between each feature block and the data distribution center as weight w 2i ;

[0126] Determine the target weight of each feature block as w i =w 1i × 2i .

[0127] It can be understood that the emotion recognition device provided in this embodiment is used to execute the emotion recognition method as described in any of the above embodiments, and achieve the same technical effects as those that can be achieved, and will not be further elaborated here.

[0128] See also Figure 3 , a certain embodiment of the present application provides a terminal device, including:

[0129] one or more processors;

[0130] A memory, coupled to the processor, for storing one or more programs;

[0131] When the one or more programs are executed by the one or more processors, the one or more processors implement the emotion recognition method as described above.

[0132] The processor is used to control the overall operation of the terminal device to complete all or part of the steps of the above-mentioned emotion recognition method. The memory is used to store various types of data to support the operation of the terminal device, and these data may include, for example, instructions for any application or method used to operate on the terminal device, as well as application-related data. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0133] In an exemplary embodiment, the terminal device can be implemented by one or more application specific integrated circuits (Application Specific 1n Integrated Circuit, referred to as ASC), digital signal processors (Digital Signal Processor, referred to as DSP), digital signal processing devices (Digital Signal Processing Device, referred to as DSPD), programmable logic devices (Programmable Logic Device, referred to as PLD), field programmable gate arrays (Field Programmable Gate Array, referred to as FPGA), controllers, microcontrollers, microprocessors or other electronic components, and is used to execute the emotion recognition method described in any of the above embodiments and achieve the same technical effect as the above method.

[0134] In another exemplary embodiment, a computer-readable storage medium including a computer program is also provided, and when the computer program is executed by a processor, the steps of the emotion recognition method described in any of the above embodiments are implemented. For example, the computer-readable storage medium can be the above-mentioned memory including the computer program, and the above-mentioned computer program can be executed by the processor of the terminal device to complete the emotion recognition method described in any of the above embodiments, and achieve the technical effect consistent with the above-mentioned method.

[0135] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications are also considered to be within the scope of protection of the present application.

Claims

1. An emotion recognition method, characterized in that: include: Based on the input continuous frame video sequence, the face area of ​​each frame image is extracted as a training sample; Input the training samples into the 3D convolutional neural network model to obtain the global feature map of each video frame sequence; The global feature map is divided into blocks, the generated local feature map and the global feature map are used as feature blocks, and the one-dimensional features of each feature block are extracted; Input the one-dimensional features into the fully connected layer of the three-dimensional convolutional neural network model and output the initial predicted sentiment score; The mean square error loss function is used to determine the loss function of each feature block, the total loss function is calculated, and the total loss function threshold is set. The three-dimensional convolutional neural network model is trained using training samples until the model converges, and the initial weight w of each feature block is determined at this time. 1i ; Based on the one-dimensional features of each feature block, the k-means algorithm is used to determine the data distribution center of the one-dimensional features of each feature block, and the Euclidean distance between each feature block and the data distribution center is defined as the weight w 2i , according to the initial weight of each feature block and the Euclidean distance, the target weight of each feature block is determined as w i =w 1i × 2i ; The initial predicted emotion score is weighted by the target weight to generate a target predicted emotion score; wherein the generated target predicted emotion score satisfies the following formula Said represents the initial predicted sentiment score; i Represents the target weight.

2. The emotion recognition method according to claim 1, characterized in that: Before extracting the face region of each frame image as a training sample, the method includes: Extract the face area of ​​each frame image and get the corresponding rectangular frame coordinates and confidence: B i ,PR i =YOLO(frame t ); In the formula, frame t represents the t-th video frame, B i Represents the coordinates of the upper left corner and lower right corner of the rectangular box marking the i-th face in the video frame, PR i Represents the confidence of the i-th face; Get the rectangular frame area with the highest confidence in each frame image, and use bilinear interpolation to process all rectangular frame areas to the same size.

3. The emotion recognition method according to claim 2, characterized in that: The step of extracting the face region of each frame of image comprises: The YOLOv3 model pre-trained with the WIDER Face dataset is used to extract the face area of ​​each frame.

4. The emotion recognition method according to claim 1, characterized in that: The step of extracting one-dimensional features of each feature block includes: A one-dimensional convolutional neural network model is used to extract one-dimensional features of each feature block.

5. The emotion recognition method according to claim 1, characterized in that: The step of determining the initial weight of each feature block includes: The mean square error loss function is used to calculate the loss of each feature block. Suppose the loss calculated for the feature blocks of the five parts is L i , i = 0, 1, 2, 3, 4, corresponding to the overall features and four local features respectively, then the final total loss function is: L = L0 + λ (L1 + L2 + L3 + L4); Among them, λ is the weight parameter that balances the overall features and local features; According to the calculation formula of the above loss function, the threshold of the loss function is set, and the three-dimensional convolutional neural network model is trained using training samples until the loss function of the model output predicting the emotion score reaches the threshold of the loss function. The model converges, and the loss value of each feature block after convergence is calculated. The loss value is inversely counted and normalized to obtain the weight parameters of the feature blocks of each part, and the weight parameters are used as the initial weights of each feature block.

6. The emotion recognition method according to claim 5, characterized in that: The initial weight of each feature block is: In the formula, l i is the loss function of the ith feature block, w 1i is the weight of the i-th feature block.

7. An emotion recognition device, characterized in that: include: A training sample acquisition unit, used for extracting a face region of each frame image as a training sample based on an input continuous frame video sequence; A global feature map acquisition unit, used for inputting training samples into a three-dimensional convolutional neural network model to obtain a global feature map of each video frame sequence; A feature map block unit is used to block the global feature map, use the generated local feature map and the global feature map as feature blocks, and extract one-dimensional features of each feature block; The target weight calculation unit is used to input the one-dimensional feature into the fully connected layer of the three-dimensional convolutional neural network model, output the initial predicted emotion score, and use the mean square error loss function to determine the loss function of each feature block, calculate the total loss function, and set the total loss function threshold. The three-dimensional convolutional neural network model is trained with training samples until the model converges, and the initial weight w of each feature block is determined at this time. 1i Based on the one-dimensional features of each feature block, the k-means algorithm is used to determine the data distribution center of the one-dimensional features of each feature block, and the Euclidean distance between each feature block and the data distribution center is defined as the weight w 2i , according to the initial weight of each feature block and the Euclidean distance, the target weight of each feature block is determined as w i =w 1i × 2i ; The emotion recognition unit is used to weight the initial predicted emotion score using the target weight to generate a target predicted emotion score; wherein the generated target predicted emotion score satisfies the following formula Said represents the initial predicted sentiment score; i Represents the target weight.

8. A terminal device, characterized in that: include: one or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the emotion recognition method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the emotion recognition method according to any one of claims 1 to 6 is implemented.

Images