Emotion recognition method, device and storage medium

Abstract

The present invention provides an emotion recognition method, device and storage medium, belonging to the field of picture recognition technology, the method comprises: S1: importing multiple emotion sample pictures, screening and analyzing the multiple emotion sample pictures respectively, and obtaining difficult sample pictures; S2: based on a convolutional neural network, extracting the features of the difficult sample pictures respectively through the convolutional neural network, and obtaining difficult sample features; S3: based on an emotion classification model, training the emotion classification model according to the multiple difficult sample features, and obtaining an emotion recognition model; S4: inputting the multiple difficult sample features into the emotion recognition model for recognition, and obtaining a recognition result. The present invention improves the accuracy of the classification model in complex emotion recognition, and at the same time, it is also helpful in the emotion recognition of compound samples.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of picture recognition, and in particular to an emotion recognition method, device and storage medium. Background Art

[0002] At present, the main indicator of emotion recognition is the recognition accuracy. Human emotions are changeable. The current emotion recognition technology can only recognize surface emotion pictures for existing emotion recognition models, but cannot accurately recognize some complex emotion pictures, such as crying from excitement or laughing from sadness. Existing emotion recognition models are difficult to accurately recognize these complex emotions, so how to effectively classify these complex emotions has become an urgent problem that needs to be solved. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an emotion recognition method, device and storage medium in view of the deficiencies in the prior art.

[0004] The present invention solves the above technical problem with the following technical solution: an emotion recognition method, comprising the following steps:

[0005] S1: importing a plurality of emotion sample pictures, screening and analyzing the plurality of emotion sample pictures respectively, and obtaining difficult sample pictures corresponding to the emotion sample pictures;

[0006] S2: Based on a convolutional neural network, extracting features of the difficult sample images from the plurality of difficult sample images respectively through the convolutional neural network to obtain difficult sample features corresponding to the difficult sample images;

[0007] S3: Based on the emotion classification model, the emotion classification model is trained according to the plurality of difficult sample features to obtain an emotion recognition model;

[0008] S4: Inputting the plurality of difficult sample features into the emotion recognition model for recognition to obtain a recognition result.

[0009] Another technical solution of the present invention to solve the above technical problem is as follows: an emotion recognition device, comprising:

[0010] A screening and analysis module, used for importing a plurality of emotion sample pictures, and screening and analyzing the plurality of emotion sample pictures respectively, to obtain difficult sample pictures corresponding to the emotion sample pictures;

[0011] A feature extraction module, configured to extract features of the difficult sample images from the plurality of difficult sample images respectively based on a convolutional neural network, and obtain difficult sample features corresponding to the difficult sample images;

[0012] A model training module, used to train the emotion classification model based on the emotion classification model according to the plurality of difficult sample features to obtain an emotion recognition model;

[0013] The recognition result obtaining module is used to input the plurality of difficult sample features into the emotion recognition model for recognition to obtain a recognition result.

[0014] Another technical solution of the present invention to solve the above technical problem is as follows: an emotion recognition device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the emotion recognition method described above is implemented.

[0015] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the emotion recognition method as described above is implemented.

[0016] The beneficial effects of the present invention are as follows: difficult sample pictures are obtained by screening and analyzing multiple emotion sample pictures respectively, difficult sample picture features of multiple difficult sample pictures are extracted respectively through a convolutional neural network to obtain difficult sample features, an emotion classification model is trained according to the multiple difficult sample features to obtain an emotion recognition model, and the multiple difficult sample features are input into the emotion recognition model to obtain a recognition result, thereby improving the accuracy of the classification model in recognizing complex emotions, and at the same time, it is also helpful in the emotion recognition of composite samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of an emotion recognition method provided by an embodiment of the present invention;

[0018] Figure 2 A module block diagram of an emotion recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0020] Figure 1 A flowchart of an emotion recognition method provided by an embodiment of the present invention.

[0021] like Figure 1 As shown, an emotion recognition method comprises the following steps:

[0022] S1: importing a plurality of emotion sample pictures, screening and analyzing the plurality of emotion sample pictures respectively, and obtaining difficult sample pictures corresponding to the emotion sample pictures;

[0023] S2: Based on a convolutional neural network, extracting features of the difficult sample images from the plurality of difficult sample images respectively through the convolutional neural network to obtain difficult sample features corresponding to the difficult sample images;

[0024] S3: Based on the emotion classification model, the emotion classification model is trained according to the plurality of difficult sample features to obtain an emotion recognition model;

[0025] S4: Inputting the plurality of difficult sample features into the emotion recognition model for recognition to obtain a recognition result.

[0026] It should be understood that the emotion sample pictures are obtained from a public emotion recognition dataset.

[0027] It should be understood that the Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure, and is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure, so they are also called "Shift-Invariant Artificial Neural Networks (SIANN)".

[0028] It should be understood that the emotion classification model is a classification model composed of an embedding layer, a convolution layer, a pooling layer, a fully connected layer and a classification layer. The essence of the emotion classification model is to convert the emotional information into a feature vector, then perform convolution calculation on the feature vector to obtain a score, and then input the score into the classification layer to obtain the emotion classification result.

[0029] In the above embodiment, difficult sample pictures are obtained by screening and analyzing multiple emotion sample pictures respectively, and difficult sample picture features of the multiple difficult sample pictures are extracted respectively through a convolutional neural network to obtain difficult sample features. The emotion classification model is trained according to the multiple difficult sample features to obtain an emotion recognition model, and the multiple difficult sample features are input into the emotion recognition model to obtain a recognition result, which improves the accuracy of the classification model in recognizing complex emotions and is also helpful in the emotion recognition of composite samples.

[0030] Optionally, as an embodiment of the present invention, in step S1, the process of screening and analyzing the plurality of emotion sample pictures respectively to obtain difficult sample pictures corresponding to the emotion sample pictures includes:

[0031] Extracting features of the emotion sample pictures from the plurality of emotion sample pictures respectively by using the convolutional neural network to obtain emotion sample features corresponding to the emotion sample pictures;

[0032] Using the emotion classification model, the emotion sample features are respectively classified to obtain a first classification probability corresponding to the emotion sample picture;

[0033] Importing a label corresponding to the emotion sample picture, respectively calculating a plurality of first classification probabilities and loss values ​​of the label corresponding to the first classification probabilities, and obtaining a first loss value corresponding to the emotion sample picture;

[0034] When the first loss value is greater than a preset difficult sample judgment value, the emotional sample image corresponding to the first loss value is used as a difficult sample image.

[0035] It should be understood that the labels may be manually annotated to verify whether the results of the classification process are correct.

[0036] It should be understood that the preset difficult sample judgment value is obtained through sample classification experiments. When the first loss value is greater than 70%, it is defined as a difficult sample, and when it is less than 70%, it is defined as a simple sample.

[0037] It should be understood that first, a convolutional neural network (i.e., the convolutional neural network) is built, and pictures in the public emotion recognition data set (i.e., the emotion sample pictures) are input into the convolutional neural network (i.e., the convolutional neural network) to extract features, and the features (i.e., the emotion sample picture features) are input into the emotion classification model (i.e., the emotion classification model) to output the classification result (i.e., the first classification probability), and then the result (i.e., the first classification probability) and the label (i.e., the true label) are subjected to loss. If the loss (i.e., the first loss value) is greater than a set threshold (i.e., the preset difficult sample judgment value), the picture (i.e., the emotion sample picture) is defined as a difficult sample (i.e., the difficult sample picture), and a set V = {v1, v2, ..., v i} is a difficult sample data set, v i Represents difficult sample images.

[0038] In the above embodiment, the difficult sample images are obtained by screening and analyzing a plurality of emotion sample images respectively, and the images of complex emotions can be screened out preferentially, providing a data basis for subsequent processing and improving the accuracy of complex emotion recognition.

[0039] Optionally, as an embodiment of the present invention, the process of step S3 includes:

[0040] Decoding the plurality of difficult sample features respectively through a GRU gated recurrent unit to obtain a plurality of emotion scores corresponding to the difficult sample features;

[0041] Respectively screening the maximum values ​​of a plurality of emotion scores corresponding to the difficult sample features, and obtaining the maximum emotion score corresponding to the difficult sample features after screening;

[0042] Performing classification processing on the maximum emotion scores of the plurality of maximum emotion scores respectively through the emotion classification model to obtain classification probabilities corresponding to the difficult sample features;

[0043] Calculating the loss values ​​of the plurality of classification probabilities respectively to obtain a second loss value corresponding to the difficult sample feature;

[0044] A plurality of the second loss values ​​are judged and analyzed to obtain an emotion recognition model.

[0045] It should be understood that the GRU gated recurrent unit is a gating mechanism in a recurrent neural network (RNN), similar to other gating mechanisms, which aims to solve the gradient vanishing / exploding problem in a standard RNN while retaining the long-term information of the sequence. GRU is as good as LSTM in many sequence tasks such as speech recognition, but it has fewer parameters than LSTM, including only a reset gate and an update gate.

[0046] It should be understood that the feature (i.e., the difficult sample feature) is input into the GRU (gated recurrent unit) decoding end to obtain multiple emotion scores Ei corresponding to the sample vi (i.e., the difficult sample feature), and the largest score Emax is selected from all the emotion scores ei and input into the emotion classification model (i.e., the emotion classification model) to obtain the classification probability Pi.

[0047] In the above embodiment, the emotion classification model is trained by using multiple difficult sample features to obtain an emotion recognition model, which improves the accuracy of the classification model in recognizing complex emotions and is also helpful in recognizing emotions of complex samples.

[0048] Optionally, as an embodiment of the present invention, the process of performing classification processing on the maximum emotion scores of the plurality of maximum emotion scores respectively through the emotion classification model to obtain the classification probability corresponding to the difficult sample feature includes:

[0049] The maximum emotion scores are calculated for the plurality of the maximum emotion scores respectively through the first formula to obtain the classification probability corresponding to the difficult sample feature, and the first formula is:

[0050] Pi = Eimax / 100,

[0051] Among them, Pi is the classification probability corresponding to the i-th difficult sample image, and Eimax is the maximum emotion score corresponding to the i-th difficult sample image.

[0052] It should be understood that the range of the emotion score is 0-100, with every 25 points being one emotion category.

[0053] In the above embodiment, the classification probability is obtained by respectively calculating the maximum emotion scores of multiple maximum emotion scores through the first formula, which provides data support for subsequent processing, can intuitively understand the accuracy of emotion recognition, and improves the accuracy of the classification model for complex emotion recognition.

[0054] Optionally, as an embodiment of the present invention, the process of respectively calculating the loss values ​​of the plurality of classification probabilities to obtain a second loss value corresponding to the difficult sample feature includes:

[0055] The loss values ​​of the plurality of classification probabilities are calculated respectively by the second formula to obtain a second loss value corresponding to the difficult sample feature. The second formula is:

[0056] FLi=-(1-Pi) γ log(Pi),

[0057] Among them, FLi is the second loss value corresponding to the i-th difficult sample image, Pi is the classification probability corresponding to the i-th difficult sample feature, γ is a parameter, γ>1, and γ is an integer.

[0058] It should be understood that after Pi (i.e., the classification probability) is transformed by the function (i.e., the second formula), the loss (i.e., the second loss value) for distinguishing difficult samples is obtained, and the loss is fed back to the emotion classification model (i.e., the emotion classification model) to update the model parameters in the direction of reducing the loss.

[0059] In the above embodiment, the second loss value is obtained by respectively calculating the loss values ​​of multiple classification probabilities through the second formula, which is beneficial to the loss of complex emotion samples and further improves the accuracy of the classification model in recognizing complex emotions.

[0060] Optionally, as an embodiment of the present invention, the process of performing judgment and analysis on the plurality of second loss values ​​to obtain the emotion recognition model includes:

[0061] Determine whether all the second loss values ​​are less than the preset emotion recognition threshold. If so, use the emotion classification model as the emotion recognition model; if not, update the parameters of the emotion classification model according to multiple second loss values, and return to step S2.

[0062] It should be understood that steps S2 and S3 are repeated until all difficulty losses (i.e., the second loss value) are less than a set threshold (i.e., the preset emotion recognition threshold), then the training of the emotion classification model (i.e., the first emotion classification model) is stopped, at which time the optimal emotion classification model (i.e., the emotion recognition model) is obtained, and then the features extracted in step S2 (i.e., the difficult sample features) are input into the optimal emotion classification model (i.e., the emotion recognition model).

[0063] In the above embodiment, the emotion recognition model is obtained by judging and analyzing multiple second loss values, which improves the accuracy of the classification model in recognizing complex emotions and is also helpful in the emotion recognition of composite samples.

[0064] Figure 2 A module block diagram of an emotion recognition device provided by an embodiment of the present invention.

[0065] Optionally, as another embodiment of the present invention, Figure 2 As shown, an emotion recognition device comprises:

[0066] A screening and analysis module, used for importing a plurality of emotion sample pictures, and screening and analyzing the plurality of emotion sample pictures respectively, to obtain difficult sample pictures corresponding to the emotion sample pictures;

[0067] A feature extraction module, configured to extract features of the difficult sample images from the plurality of difficult sample images respectively based on a convolutional neural network, and obtain difficult sample features corresponding to the difficult sample images;

[0068] A model training module, used to train the emotion classification model based on the emotion classification model according to the plurality of difficult sample features to obtain an emotion recognition model;

[0069] The recognition result obtaining module is used to input the plurality of difficult sample features into the emotion recognition model for recognition to obtain a recognition result.

[0070] Optionally, as an embodiment of the present invention, the screening and analysis module is used to:

[0071] Extracting features of the emotion sample pictures from the plurality of emotion sample pictures respectively by using the convolutional neural network to obtain emotion sample features corresponding to the emotion sample pictures;

[0072] Using the emotion classification model, the emotion sample features are respectively classified to obtain a first classification probability corresponding to the emotion sample picture;

[0073] Importing a label corresponding to the emotion sample picture, respectively calculating a plurality of first classification probabilities and loss values ​​of the label corresponding to the first classification probabilities, and obtaining a first loss value corresponding to the emotion sample picture;

[0074] When the first loss value is greater than a preset difficult sample judgment value, the emotional sample image corresponding to the first loss value is used as a difficult sample image.

[0075] Optionally, another embodiment of the present invention provides an emotion recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the emotion recognition method described above is implemented. The device may be a computer or other device.

[0076] Optionally, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the emotion recognition method as described above is implemented.

[0077] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0081] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An emotion recognition method, characterized in that: The steps include: S1: importing a plurality of emotion sample pictures, screening and analyzing the plurality of emotion sample pictures respectively, and obtaining difficult sample pictures corresponding to the emotion sample pictures; S2: Based on a convolutional neural network, extracting features of the difficult sample images from the plurality of difficult sample images respectively through the convolutional neural network to obtain difficult sample features corresponding to the difficult sample images; S3: Based on the emotion classification model, the emotion classification model is trained according to the plurality of difficult sample features to obtain an emotion recognition model; S4: inputting the plurality of difficult sample features into the emotion recognition model for recognition to obtain a recognition result; In step S1, the process of screening and analyzing the plurality of emotion sample pictures respectively to obtain difficult sample pictures corresponding to the emotion sample pictures includes: Extracting features of the emotion sample pictures from the plurality of emotion sample pictures respectively by using the convolutional neural network to obtain emotion sample features corresponding to the emotion sample pictures; Using the emotion classification model, the emotion sample features are respectively classified to obtain a first classification probability corresponding to the emotion sample picture; Importing a label corresponding to the emotion sample picture, respectively calculating a plurality of first classification probabilities and loss values ​​of the label corresponding to the first classification probabilities, and obtaining a first loss value corresponding to the emotion sample picture; When the first loss value is greater than a preset difficult sample judgment value, the emotional sample image corresponding to the first loss value is used as a difficult sample image; The process of step S3 includes: Decoding the plurality of difficult sample features respectively through a GRU gated recurrent unit to obtain a plurality of emotion scores corresponding to the difficult sample features; Respectively screening the maximum values ​​of a plurality of emotion scores corresponding to the difficult sample features, and obtaining the maximum emotion score corresponding to the difficult sample features after screening; Performing classification processing on the maximum emotion scores of the plurality of maximum emotion scores respectively through the emotion classification model to obtain classification probabilities corresponding to the difficult sample features; Calculating the loss values ​​of the plurality of classification probabilities respectively to obtain a second loss value corresponding to the difficult sample feature; A plurality of the second loss values ​​are judged and analyzed to obtain an emotion recognition model.

2. The emotion recognition method according to claim 1, characterized in that: The process of performing classification processing on the maximum emotion scores of the plurality of maximum emotion scores respectively through the emotion classification model to obtain the classification probability corresponding to the difficult sample feature includes: The maximum emotion scores are calculated for the plurality of the maximum emotion scores respectively through the first formula to obtain the classification probability corresponding to the difficult sample feature, and the first formula is: Pi = Eimax / 100, Among them, Pi is the classification probability corresponding to the i-th difficult sample image, and Eimax is the maximum emotion score corresponding to the i-th difficult sample image.

3. The emotion recognition method according to claim 1, characterized in that: The process of respectively calculating the loss values ​​of the plurality of classification probabilities to obtain a second loss value corresponding to the difficult sample feature includes: The loss values ​​of the plurality of classification probabilities are calculated respectively by the second formula to obtain a second loss value corresponding to the difficult sample feature. The second formula is: FLi=-(1-Pi) γ log(Pi), Among them, FLi is the second loss value corresponding to the i-th difficult sample image, Pi is the classification probability corresponding to the i-th difficult sample feature, γ is a parameter, γ>1, and γ is an integer.

4. The emotion recognition method according to claim 1, characterized in that: The process of performing judgment and analysis on the plurality of second loss values ​​to obtain the emotion recognition model comprises: Determine whether all the second loss values ​​are less than the preset emotion recognition threshold. If so, use the emotion classification model as the emotion recognition model; if not, update the parameters of the emotion classification model according to multiple second loss values, and return to step S2.

5. An emotion recognition device, characterized in that: include: A screening and analysis module, used for importing a plurality of emotion sample pictures, and screening and analyzing the plurality of emotion sample pictures respectively, to obtain difficult sample pictures corresponding to the emotion sample pictures; A feature extraction module, configured to extract features of the difficult sample images from the plurality of difficult sample images respectively based on a convolutional neural network, and obtain difficult sample features corresponding to the difficult sample images; A model training module, used to train the emotion classification model based on the emotion classification model according to the plurality of difficult sample features to obtain an emotion recognition model; A recognition result obtaining module, used for inputting the plurality of difficult sample features into the emotion recognition model for recognition to obtain a recognition result; The screening analysis module is used for: Extracting features of the emotion sample pictures from the plurality of emotion sample pictures respectively by using the convolutional neural network to obtain emotion sample features corresponding to the emotion sample pictures; Using the emotion classification model, the emotion sample features are respectively classified to obtain a first classification probability corresponding to the emotion sample picture; Importing a label corresponding to the emotion sample picture, respectively calculating a plurality of first classification probabilities and loss values ​​of the label corresponding to the first classification probabilities, and obtaining a first loss value corresponding to the emotion sample picture; When the first loss value is greater than a preset difficult sample judgment value, the emotional sample image corresponding to the first loss value is used as a difficult sample image; The model training module is specifically used for: Decoding the plurality of difficult sample features respectively through a GRU gated recurrent unit to obtain a plurality of emotion scores corresponding to the difficult sample features; Respectively screening the maximum values ​​of a plurality of emotion scores corresponding to the difficult sample features, and obtaining the maximum emotion score corresponding to the difficult sample features after screening; Performing classification processing on the maximum emotion scores of the plurality of maximum emotion scores respectively through the emotion classification model to obtain classification probabilities corresponding to the difficult sample features; Calculating the loss values ​​of the plurality of classification probabilities respectively to obtain a second loss value corresponding to the difficult sample feature; A plurality of the second loss values ​​are judged and analyzed to obtain an emotion recognition model.

6. An emotion recognition system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the emotion recognition method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the emotion recognition method according to any one of claims 1 to 4 is implemented.

Images