Emotion recognition model training method, emotion recognition method, device, and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a training method of an emotion recognition model, an emotion recognition method, equipment and a medium. The training method comprises the following steps: acquiring a sample face image; performing region extraction on the sample face image to obtain a first region image containing an eye part in the sample face image and a second region image containing a mouth part in the sample face image; performing feature replacement on the first region image or the second region image and performing feature fusion based on a feature replacement result to obtain first fused features; inputting the first fused features into a preset to-be-trained model to obtain a training recognition result; and training the to-be-trained model based on the training recognition result to obtain an emotion recognition model. The embodiment of the application can improve the accuracy of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a training method of an emotion recognition model, an emotion recognition method, equipment and a medium. BACKGROUND

[0002] At present, in many business fields, in order to better carry out business, it is often necessary to understand the emotional state of the user, and for this, the related art often judges the emotional state of the user based on the subjective experience of the staff, which can cause a large emotional misjudgment and affect the accuracy of emotion recognition. Therefore, how to improve the accuracy of emotion recognition has become a technical problem to be solved. SUMMARY

[0003] The purpose of the present application is to provide a training method of an emotion recognition model, an emotion recognition method, equipment and a medium, which aims to improve the accuracy of emotion recognition.

[0004] The present application provides a training method of an emotion recognition model, comprising:

[0005] Obtaining a sample face image;

[0006] Region extraction is performed on the sample face image to obtain a first region image containing the eye part in the sample face image and a second region image containing the mouth part in the sample face image;

[0007] Feature replacement is performed on the first region image or the second region image, and feature fusion is performed based on the feature replacement result to obtain a first fusion feature;

[0008] The first fusion feature is input into a preset to-be-trained model to obtain a training recognition result;

[0009] The to-be-trained model is trained based on the training recognition result to obtain an emotion recognition model.

[0010] In some embodiments, the region extraction on the sample face image to obtain the first region image containing the eye part in the sample face image and the second region image containing the mouth part in the sample face image comprises:

[0011] Based on the face contour feature, image segmentation is performed on the sample face image to obtain a first segmentation image and a second segmentation image;

[0012] Region of interest extraction is performed on the first segmentation image and the second segmentation image to obtain the first region image and the second region image.

[0013] In some embodiments, before the feature replacement on the first region image or the second region image and the feature fusion based on the feature replacement result to obtain the first fusion feature, the method further comprises:

[0014] inputting the first region image into a pre-trained glasses removal network to perform mask operation on the glasses feature and the glasses shadow feature, to obtain the first region image after removing the glasses; the glasses removal network is trained based on an adversarial learning mechanism.

[0015] In some embodiments, the feature replacement on the first region image or the second region image and the feature fusion based on the feature replacement result to obtain the first fusion feature comprises:

[0016] replacing the first region image with a first replacement image, or replacing the second region image with a second replacement image; the first replacement image contains the eye part of the face in a preset replacement image, and the second replacement image contains the mouth part of the face in a preset replacement image;

[0017] performing feature extraction on the first region image and the second replacement image, or performing feature extraction on the first replacement image and the second region image, to obtain a first feature vector of the feature information of the eye part of the face and a second feature vector of the feature information of the mouth part of the face;

[0018] performing attention mechanism processing on the first feature vector and the second feature vector to obtain spatial attention weight features and channel attention weight features of both the first feature vector and the second feature vector;

[0019] performing feature concatenation on the spatial attention weight features and the channel attention weight features and performing feature mapping based on the feature concatenation result to obtain the first fusion feature.

[0020] In some embodiments, the inputting of the first fusion feature into the to-be-trained model to obtain a training recognition result comprises:

[0021] mapping the first fusion feature to a preset mapping space to obtain a mapping feature;

[0022] determining a feature similarity between the mapping feature and a sample feature;

[0023] performing multi-label classification on the mapping feature according to the feature similarity to obtain the training recognition result.

[0024] In some embodiments, the training of the to-be-trained model based on the training recognition result to obtain an emotion recognition model comprises:

[0025] determining model loss information, wherein the model loss information represents a deviation between the training recognition result and an expected recognition result determined by a preset label of the sample face image;

[0026] determining whether the model loss information is in a loss threshold interval;

[0027] if not, adjusting network parameters of the to-be-trained model, and returning to the step of inputting the first fusion feature into the to-be-trained model to obtain a training recognition result;

[0028] if yes, determining whether a training reset number reaches a reset threshold;

[0029] if not, returning to the step of obtaining a sample face image;

[0030] if yes, ending training to obtain the emotion recognition model.

[0031] The embodiment of the application further provides an emotion recognition method, comprising:

[0032] obtaining a to-be-recognized face image;

[0033] performing region extraction on the to-be-recognized face image to obtain a third region image containing an eye part in the to-be-recognized face image and a fourth region image containing a mouth part in the to-be-recognized face image;

[0034] performing feature fusion on the third region image and the fourth region image to obtain a second fusion feature;

[0035] inputting the second fusion feature into an emotion recognition model to obtain an emotion recognition result, wherein the emotion recognition model is trained by the emotion recognition model training method.

[0036] In some embodiments, before the step of performing feature fusion on the third region image and the fourth region image to obtain a second fusion feature, the method further comprises:

[0037] inputting the third region image into a pre-trained glasses removal network to perform a mask operation on a glasses feature and a glasses shadow feature, to obtain a third region image after glasses removal, wherein the glasses removal network is trained based on an adversarial learning mechanism.

[0038] The embodiment of the application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0039] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method.

[0040] The present application has the following advantages: the first region image containing the eye part in the sample face image and the second region image containing the mouth part in the sample face image are extracted from the sample face image as the basis for emotion recognition, feature replacement is performed on the first region image or the second region image, and feature fusion is performed based on the feature replacement result to obtain first fusion features, the first fusion features are input into a to-be-trained model, the to-be-trained model is trained by using a training recognition result obtained by the to-be-trained model, and an emotion recognition model is obtained after the training is completed. Since the first fusion features are obtained by performing feature fusion on the first region image or the second region image after feature replacement, the to-be-trained model is trained by using the training recognition result obtained by the to-be-trained model, the to-be-trained model can learn the degree of dependence of different emotions on the eye part and the mouth part in the training process, and thus the emotion recognition model trained can accurately recognize the emotion category of the face in the to-be-recognized face image. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 FIG. 1 is a flowchart of an emotion recognition model training method provided by an embodiment of the present application.

[0042] Figure 2 FIG. 2 is a flowchart of a specific method of step S103 provided by an embodiment of the present application.

[0043] Figure 3 FIG. 3 is a flowchart of a specific method of step S104 provided by an embodiment of the present application.

[0044] Figure 4 FIG. 4 is a flowchart of a specific method of step S105 provided by an embodiment of the present application.

[0045] Figure 5 FIG. 5 is a flowchart of an emotion recognition method provided by an embodiment of the present application.

[0046] Figure 6 FIG. 6 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0048] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical sequence is shown in the flowchart, in some cases, the steps shown can be performed in a manner different from the module division in the device or the sequence in the flowchart. The terms "first", "second", and the like in the description and claims and drawings are used to distinguish similar objects, and are not intended to describe a specific order or sequence.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification herein is for the purpose of describing the embodiments of the present application only and is not intended to be limiting of the present application.

[0050] The relevant data collection and processing in the embodiments of the present application should be strictly in accordance with the requirements of relevant laws and regulations, and in line with social morality and public interest, obtaining the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carrying out subsequent data use and processing.

[0051] The embodiments of the present application provide a training method of an emotion recognition model.

[0052] Figure 1 is a flowchart of the training method of the emotion recognition model provided by the embodiments of the present application. Referring to Figure 1 In some embodiments, the method includes but is not limited to steps S101 to S105.

[0053] It should be noted that the execution subject of the training method of the emotion recognition model provided by the embodiments of the present application can be applied to an end-side device, or can be applied to a cloud-side device, and the embodiments do not limit this.

[0054] Step S101, obtaining a sample face image.

[0055] The sample face image is an image containing at least one face.

[0056] The execution subject can obtain a sample face image obtained by shooting a face, which can be a sample face image obtained by shooting multiple faces at different time periods and / or different locations.

[0057] In some embodiments, a data acquisition module can be used to collect face images, and the face images can be collected at a frame rate of no less than 200 fps to obtain a sample image sequence composed of multiple sample face images.

[0058] In some embodiments, after obtaining the sample face image, the subject pre-processes the sample face image. In specific implementations, the pre-processing of the sample face image includes image cropping, normalization, motion magnification processing and the like. The image cropping can be to identify and remove irrelevant background by a face detection algorithm, and only keep the facial region. The normalization can be to adjust the image brightness and contrast to a suitable range. The motion magnification processing can be to generate enlarged frames, On-A frames and Off-A frames by using a Learning-based Motion Magnification algorithm to highlight the dynamic change characteristics of expressions.

[0059] In step S102, a region extraction is performed on the sample face image to obtain a first region image containing the eye part in the sample face image and a second region image containing the mouth part in the sample face image.

[0060] It can be understood that the first region image and the second region image are sub-images of the sample face image, and the first region image containing the eye part in the sample face image and the second region image containing the mouth part in the sample face image are separated by performing region extraction on the sample face image.

[0061] In some embodiments, the subject performing the region extraction on the sample face image can use a region of interest extraction method to extract the region of interest from the sample face image according to a preset region of interest extraction logic to obtain the first region image containing the eye part in the sample face image and the second region image containing the mouth part in the sample face image.

[0062] In step S103, a feature replacement is performed on the first region image or the second region image, and a feature fusion is performed based on the feature replacement result to obtain a first fused feature.

[0063] It can be understood that the feature replacement refers to replacing a specific feature in an image or directly replacing the image, for example, replacing the first region image with another region image, and the feature fusion refers to splicing and fusing specific features in two or more images in a preset combination manner.

[0064] In some embodiments, the subject performing the feature replacement on the first region image or the feature replacement on the second region image can replace the first region image or the second region image with another region image, or replace the eye part in the first region image with another form of eye part or replace the mouth part in the second region image with another form of mouth part to obtain a feature replacement result. Then, a feature extraction is performed on the two sets of region images after the feature replacement, and a feature fusion is performed on the image features obtained by the feature extraction to obtain the first fused feature.

[0065] Step S104, input the first fusion feature into a preset to-be-trained model to obtain a training recognition result.

[0066] The training recognition result is obtained by the to-be-trained model based on the feature information of the eyes and the mouth in the first fusion feature for emotion prediction, and is used to represent the emotion type corresponding to the face in the sample face image.

[0067] In some embodiments, the execution subject inputs the first fusion feature into the preset to-be-trained model, and the to-be-trained model performs emotion prediction based on the feature information of the eyes and the mouth in the first fusion feature to obtain the training recognition result. In a specific implementation, the to-be-trained model maps the input first fusion feature to a high-dimensional space to obtain a high-dimensional feature representation of the first fusion feature, then performs attention mechanism processing on the high-dimensional feature representation to obtain an attention score of each emotion category corresponding to the first fusion feature, and further performs similarity evaluation on the first fusion feature and the corresponding emotion category based on the attention score of each emotion category corresponding to the first fusion feature, and outputs the training recognition result based on the similarity evaluation result.

[0068] Step S105, train the to-be-trained model based on the training recognition result to obtain an emotion recognition model.

[0069] In some embodiments, the execution subject inputs the first fusion feature obtained from the multiple sets of sample face images into the to-be-trained model, and adjusts the network parameters of the to-be-trained model in combination with the preset label of the sample face image each time the training recognition result is obtained, and finally obtains the emotion recognition model. In a specific implementation, the multiple sample face images are traversed, and the to-be-trained model is used to generate a training recognition result according to the first fusion feature obtained from the sample face image to iteratively train the to-be-trained model until the model loss information meets the training end condition, and the emotion recognition model is obtained after the training is completed.

[0070] In summary, the training method of the emotion recognition model provided in the embodiments of the present application extracts a first region image containing the eye part in the sample face image and a second region image containing the mouth part in the sample face image from the sample face image as the basis for emotion recognition, performs feature replacement on the first region image or the second region image, and performs feature fusion based on the feature replacement result to obtain first fused features, and inputs the first fused features into a to-be-trained model, trains the to-be-trained model by using the training recognition result obtained by the to-be-trained model, and obtains an emotion recognition model after the training is completed. Since the first fused features are obtained by performing feature fusion after feature replacement on the first region image or the second region image, the first fused features are input into the to-be-trained model, and the to-be-trained model is trained by using the training recognition result obtained by the to-be-trained model, the to-be-trained model can learn the degree of dependence of different emotions on the eye part and the mouth part in the training process, so that the emotion recognition model trained can accurately recognize the emotion category of the face in the to-be-recognized face image.

[0071] In some embodiments, the step S102 includes: performing image segmentation on the sample face image based on face contour features to obtain a first segmented image and a second segmented image; and performing region of interest extraction on the first segmented image and the second segmented image to obtain the first region image and the second region image.

[0072] In the embodiments of the present application, the subject performs the step S102 by using a region extraction network.

[0073] In specific implementations, the sample face image is input into the region extraction network, in which a pixel position-based segmentation method is used to divide the upper half of the sample face image as the eye region and the lower half of the sample face image as the mouth region, for example, the sample face image with a pixel size of 640x480 is segmented at the coordinate of 240 pixels to obtain a first segmented image and a second segmented image, and then a region of interest extraction algorithm is used to perform region of interest extraction on the first segmented image and the second segmented image to obtain the first region image and the second region image.

[0074] In some embodiments, when performing region of interest extraction on the first segmented image and the second segmented image, other regions of interest can be set in addition to the eye and mouth regions, for example, eyebrows, or regions with low correlation with emotions, for example, the chin. In this way, it can be ensured that the first region image and the second region image mainly contain regions closely related to expressions, thereby improving accuracy.

[0075] In some embodiments, before the step S103, the method further includes: inputting the first region image into a pre-trained glasses removal network to perform mask operation on the glasses features and the glasses shadow features to obtain the first region image after the glasses are removed.

[0076] In the embodiment of the present application, when the face in the first region image contains eyeglass features, the first region image is input into the pre-trained eyeglass removal network, and the first region image is subjected to mask operation for eyeglass features and eyeglass shadow features by the eyeglass removal network to obtain the first region image after removing the eyeglasses.

[0077] In specific implementation, the eyeglass removal network is composed of an eyeglass feature removal network and an eyeglass shadow feature removal network. The eyeglass feature removal network removes the eyeglass features from the first region image by using a pre-trained eyeglass mask to set the pixel values constituting the eyeglass features in the first region image to 0, and then compares the image processed by the eyeglass mask with a real eyeglass-free image to generate an image after removing the eyeglass features. The eyeglass shadow feature removal network removes the eyeglass shadow features from the image after removing the eyeglass features by using a pre-trained eyeglass shadow mask to set the pixel values constituting the eyeglass shadow features in the image after removing the eyeglass features to 0, and then compares the image processed by the eyeglass shadow mask with a real eyeglass shadow-free image to obtain the first region image after removing the eyeglasses.

[0078] The eyeglass removal network is trained based on an adversarial learning mechanism. In some embodiments, a synthetic image set containing a large number of synthetic images containing different light, expression and eyeglass wearing states is constructed. The synthetic images are obtained by rendering synthetic data using multiple HDR light conditions. Then, the eyeglass feature removal network, the eyeglass shadow feature removal network and the corresponding discriminative model are initialized. The discriminative model is trained using the synthetic image set and a large number of real images containing different light, expression and eyeglass wearing states, so that the discriminative model can as accurately as possible determine whether the input image is a synthetic image or a real image. Then, the eyeglass feature removal network and the eyeglass shadow feature removal network are trained using the synthetic image set and a large number of real images containing different light, expression and eyeglass wearing states. The images after removing the eyeglass features and the images after removing the eyeglass shadow features are input into the trained discriminative model to determine the L1 regression loss information between the images after removing the eyeglass features and the real images without eyeglass features, and the L1 regression loss information between the images after removing the eyeglass shadow features and the real images without eyeglass shadow features. The network parameters of the eyeglass feature removal network and the eyeglass shadow feature removal network are adjusted according to the above L1 regression loss information. When the training end condition is met, the pre-trained eyeglass removal network is obtained.

[0079] In some embodiments, the synthetic image set is constructed by using preset 3D face data (such as 438 subjects' facial scan data) and preset multiple 3D glasses models, marking key nodes (such as nodes at positions of temples, nose bridge, etc.) on the glasses models and corresponding vertices on the face template model by node registration technology, then calculating rotation, translation and scaling parameters to make the glasses models naturally and accurately worn on the 3D face models, rendering the synthetic data using 367 different HDR lighting conditions to generate a large number of synthetic images containing different lightings, expressions and glasses wearing states.

[0080] Figure 2 is a flowchart of a specific method of step S103 provided by the embodiments of the present application. In some embodiments, referring to Figure 2 , the method includes but is not limited to steps S201 to S204.

[0081] Step S201 replaces the first region image with a first replacement image, or replaces the second region image with a second replacement image.

[0082] The first replacement image contains an eye part of a face in a preset replacement image, and the second replacement image contains a mouth part of a face in a preset replacement image.

[0083] It can be understood that the first replacement image and the second replacement image are region images obtained by performing region extraction on a preset replacement image, so as to obtain the first replacement image containing the eye part of the face in the preset replacement image and the second replacement image containing the mouth part of the face in the preset replacement image. The preset replacement image can be other sample face image or other face image.

[0084] In some embodiments, the execution subject can determine whether to replace the first region image or the second region image according to a preset replacement probability (for example, the replacement probability is 0.5), replace the first region image with the first replacement image when it is determined to replace the first region image, and replace the second region image with the second replacement image when it is determined to replace the second region image.

[0085] Step S202 performs feature extraction on the first region image and the second replacement image, or performs feature extraction on the first replacement image and the second region image, to obtain a first feature vector of feature information of the eye part of the face and a second feature vector of feature information of the mouth part of the face.

[0086] In the embodiments of the present application, the execution subject implements step S202 by using a feature extraction network.

[0087] In some embodiments, when the first region image is replaced by the first substitute image, the feature extraction network performs feature extraction on the first substitute image and the second region image, when the second region image is replaced by the second substitute image, the feature extraction network performs feature extraction on the first region image and the second substitute image, the feature extraction network has a base layer and a branch layer, the base layer extracts general features in the input image, and the branch layer extracts unique expression features in the input image, to obtain a first feature vector of the feature information of the eye of the face and a second feature vector of the feature information of the mouth of the face.

[0088] In step S203, the first feature vector and the second feature vector are processed by an attention mechanism to obtain spatial attention weight features and channel attention weight features of both the first feature vector and the second feature vector.

[0089] In the embodiments of the present application, the subject performs step S203 using an attention mechanism network.

[0090] In some embodiments, the attention mechanism network adopts a Shuffle Attention mechanism to process the first feature vector and the second feature vector by attention mechanism, and assigns corresponding spatial attention weight features and channel attention weight features to the first feature vector and the second feature vector through attention weight calculation, so as to enhance the attention and extraction ability of important features.

[0091] In step S204, the spatial attention weight features and the channel attention weight features are spliced and mapped based on the splicing result to obtain the first fusion feature.

[0092] In some embodiments, the spatial attention weight features and the channel attention weight features of the first feature vector are spliced, and the spatial attention weight features and the channel attention weight features of the second feature vector are spliced to obtain attention weight splicing features, i.e. the feature splicing result, and then according to the dependence of different emotions on eye and mouth features, the attention weight splicing features are assigned with corresponding weights, and the attention weight splicing features of both the first feature vector and the second feature vector are adaptively fused to obtain the first fusion feature, which can strengthen the facial expression feature expression.

[0093] Figure 3 is a flowchart of the specific method of step S104 provided by the embodiments of the present application. In some embodiments, referring to Figure 3 The method includes but is not limited to steps S301 to S303.

[0094] In step S301, the first fusion feature is mapped to a preset mapping space to obtain a mapping feature.

[0095] Step S302, determining a feature similarity between the mapping feature and the sample feature.

[0096] Step S303, performing multi-label classification on the mapping feature according to the feature similarity, to obtain a training recognition result.

[0097] In the embodiments of the present application, the execution subject implements steps S301 to S303 by using a classification network.

[0098] In some embodiments, the classification network includes a full connection layer and a classifier layer, the full connection layer includes multiple hidden layers (such as two hidden layers with 512 and 256 neurons respectively), the full connection layer maps the first fusion feature to a preset mapping space and performs activation processing on the feature of the first fusion feature mapped to the mapping space, to obtain and output the mapping feature, then an AU alignment auxiliary task is introduced, the AU similarity score between the mapping feature and the sample feature is calculated, and the similarity score calculation method is reasonably set according to the AU label combination of the mapping feature, to determine the feature similarity between the mapping feature and the sample feature, the feature similarity is input into the classifier layer, and the classifier layer performs multi-label classification on the mapping feature according to the feature similarity, to obtain the training recognition result.

[0099] Figure 4 is a flowchart of the specific method of step S105 provided by the embodiments of the present application. In some embodiments, referring to Figure 4 , the method includes but is not limited to steps S401 to S405.

[0100] Step S401, determining model loss information.

[0101] The model loss information represents the deviation between the training recognition result and an expected recognition result, and the expected recognition result is determined by a preset label of the sample face image.

[0102] Step S402, judging whether the model loss information is within a loss threshold interval.

[0103] If not, step S403 is performed; if yes, step S404 is performed.

[0104] Step S403, adjusting the network parameters of the model to be trained. Return to step S104.

[0105] Step S404, judging whether the training reset number reaches a reset threshold.

[0106] If not, step S104 is performed; if yes, step S405 is performed.

[0107] Step S405, if yes, ending the training to obtain an emotion recognition model.

[0108] In some embodiments, a loss threshold interval and a reset threshold are preset as a training end condition, when the model loss information is in the loss threshold interval and the training reset number reaches the reset threshold, the training is ended, and the last iteration of the trained model is obtained, that is, the emotion recognition model. When the training reset number does not reach the reset threshold, the network parameters of the trained model are adjusted according to the deviation of the model loss information from the loss threshold interval, so that the model loss information gradually approaches the loss threshold interval in the iteration process and finally is in the loss threshold interval, then new sample face images are obtained, the first fusion features obtained from the new sample face images are used to iteratively train the trained model, until the model loss information is in the loss threshold interval, and the above steps are repeated until the training reset number reaches the reset threshold, the training is ended, and the emotion recognition model is obtained, which can adapt the obtained emotion recognition model to the emotion prediction of various to-be-identified face images.

[0109] The embodiment of the present application provides an emotion recognition method.

[0110] Figure 5 is a flowchart of the emotion recognition method provided by the embodiment of the present application. Referring to Figure 5 In some embodiments, the method includes but is not limited to steps S501 to S504.

[0111] Step S501, obtaining a to-be-identified face image.

[0112] Step S502, performing region extraction on the to-be-identified face image to obtain a third region image containing an eye part in the to-be-identified face image and a fourth region image containing a mouth part in the to-be-identified face image.

[0113] Step S503, performing feature fusion on the third region image and the fourth region image to obtain a second fusion feature.

[0114] Step S504, inputting the second fusion feature into an emotion recognition model to obtain an emotion recognition result.

[0115] The emotion recognition model is trained by the above-mentioned emotion recognition model training method.

[0116] The emotion recognition method provided by the embodiment of the present application can accurately identify the emotion category corresponding to the face in the to-be-identified face image by performing region extraction on the to-be-identified face image to obtain a third region image containing an eye part in the to-be-identified face image and a fourth region image containing a mouth part in the to-be-identified face image, then performing feature fusion on the third region image and the fourth region image to obtain a second fusion feature, and inputting the second fusion feature into the emotion recognition model trained by the above-mentioned emotion recognition model training method.

[0117] In some embodiments, before step S503, the method further includes: inputting the third region image into a pre-trained glasses removing network to perform a mask operation on the glasses feature and the glasses shadow feature, to obtain a third region image after removing the glasses. The glasses removing network is trained based on an adversarial learning mechanism.

[0118] Figure 6 is a block diagram of an electronic device according to an example embodiment.

[0119] The electronic device 600 according to this implementation of the disclosure will be described below with reference to Figure 6 Figure 6 The displayed electronic device 600 is merely an example and should not impose any limitation on the function and scope of use of the embodiments of the disclosure.

[0120] As shown in Figure 6 , the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0121] The storage unit stores program codes that can be executed by the processing unit 610, so that the processing unit 610 performs the steps of various example embodiments according to the disclosure described in the above pile quality detection method part of the specification.

[0122] The storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 6201 and / or a cache memory unit 6202, and can further include a read-only memory (ROM) 6203.

[0123] The storage unit 620 can further include program / utilities 6204 having a set of (at least one) program modules 6205, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0124] The bus 630 can be one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0125] ​The electronic device 600 can also communicate with one or more external devices 600' such as a keyboard, a pointing device, a Bluetooth device, etc.; and can communicate with one or more devices that enable a user to interact with the electronic device 600 and / or one or more devices (e.g. routers, modems, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface 650. Still yet, the electronic device 600 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) via network adapter 660. The network adapter 660 can communicate with the other components of the electronic device 600 via bus 630. It should be understood that, although not shown explicitly, other hardware and / or software components could be used in conjunction with the electronic device 600. Such components include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0126] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the stockpile quality detection method.

[0127] The training method of the emotion recognition model, the emotion recognition method, the device and the medium provided by the embodiment of the present application extract a first region image containing an eye part in a sample face image and a second region image containing a mouth part in the sample face image from the sample face image as a basis for emotion recognition, perform feature replacement on the first region image or the second region image, and perform feature fusion based on the feature replacement result to obtain a first fusion feature, input the first fusion feature into a to-be-trained model, train the to-be-trained model by using a training recognition result obtained by the to-be-trained model, and obtain an emotion recognition model after training is completed. Since the first fusion feature is obtained by performing feature fusion after feature replacement on the first region image or the second region image, the first fusion feature is input into the to-be-trained model, and the to-be-trained model is trained by using a training recognition result obtained by the to-be-trained model, the to-be-trained model can learn the degree of dependence of different emotions on the eye part and the mouth part in the training process, so that the emotion recognition model trained can accurately recognize the emotion category of the face in the to-be-recognized face image.

[0128] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to make a computing device (which can be a personal computer, a server, or a network device, etc.) execute the above-mentioned method according to the embodiments of the present disclosure.

[0129] The program product can employ any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0130] The computer readable storage medium can include a data signal carried in a baseband or propagated as a carrier wave in a propagated data signal, in which the readable program code is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium can also be any readable medium that is not a readable storage medium, which can send, propagate, or transmit the program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0131] Those skilled in the art can understand that the above modules can be distributed in the device according to the description of the embodiments, or can be changed in one or more devices different from the embodiments. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0132] The example embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structure, arrangement or implementation method described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent arrangements within the spirit and scope of the appended claims.

Claims

1. A training method for an emotion recognition model, characterized in that, The method comprises the following steps: obtaining a sample face image; performing region extraction on the sample face image to obtain a first region image containing eye parts in the sample face image and a second region image containing mouth parts in the sample face image; performing feature replacement on the first region image or the second region image and performing feature fusion based on the feature replacement result to obtain a first fusion feature, comprising: replacing the eye parts in the first region image with other forms of eye parts or replacing the mouth parts in the second region image with other forms of mouth parts to obtain the feature replacement result, performing feature extraction on the two groups of region images after feature replacement and performing feature fusion on the image features obtained by feature extraction to obtain the first fusion feature; inputting the first fusion feature into a preset to-be-trained model to obtain a training recognition result; the training recognition result is obtained by emotion prediction based on feature information of both the eye parts and the mouth parts in the first fusion feature by the to-be-trained model, and is used to represent an emotion type corresponding to a face in the sample face image; training the to-be-trained model based on the training recognition result to obtain an emotion recognition model; before the step of performing feature replacement on the first region image or the second region image and performing feature fusion based on the feature replacement result to obtain a first fusion feature, the method further comprises the following steps: inputting the first region image into a pre-trained glasses removal network to perform mask operation on the glasses feature and the glasses shadow feature, and obtaining a first region image after removing the glasses; the glasses removal network is trained based on an adversarial learning mechanism; when the face in the first region image contains a glasses feature, the first region image is input into the glasses removal network, the mask operation is performed on the glasses feature and the glasses shadow feature in the first region image by the glasses removal network, and the first region image after removing the glasses is obtained. 2.The method of claim 1, wherein, The step of performing region extraction on the sample face image to obtain a first region image containing eye parts in the sample face image and a second region image containing mouth parts in the sample face image comprises the following steps: performing image segmentation on the sample face image based on face contour features to obtain a first segmentation image and a second segmentation image; performing region of interest extraction on the first segmentation image and the second segmentation image to obtain the first region image and the second region image. 3.The method of claim 1, wherein, The step of performing feature replacement on the first region image or the second region image and performing feature fusion based on the feature replacement result to obtain a first fusion feature comprises the following steps: replacing the first region image with a first replacement image or replacing the second region image with a second replacement image; the first replacement image contains eye parts of a face in a preset replacement image, and the second replacement image contains mouth parts of a face in a preset replacement image; performing feature extraction on the first region image and the second replacement image or performing feature extraction on the first replacement image and the second region image to obtain a first feature vector of feature information of the eye parts of the face and a second feature vector of feature information of the mouth parts of the face; perform attention mechanism processing on the first feature vector and the second feature vector to obtain spatial attention weight features and channel attention weight features of both the first feature vector and the second feature vector; perform feature splicing on the spatial attention weight features and the channel attention weight features and perform feature mapping based on a result of the feature splicing to obtain the first fusion feature. 4.The method of claim 1, wherein, The inputting of the first fusion feature into the to-be-trained model to obtain a training recognition result comprises: mapping the first fusion feature to a preset mapping space to obtain a mapped feature; determining a feature similarity between the mapped feature and a sample feature; performing multi-label classification on the mapped feature according to the feature similarity to obtain the training recognition result. 5.The method of claim 1, wherein, The training of the to-be-trained model based on the training recognition result to obtain an emotion recognition model comprises: determining model loss information; the model loss information represents a deviation between the training recognition result and an expected recognition result, and the expected recognition result is determined by a preset label of the sample face image; determining whether the model loss information is within a loss threshold interval; if not, adjusting network parameters of the to-be-trained model; and returning to the inputting of the first fusion feature into the to-be-trained model to obtain a training recognition result; if yes, determining whether a training reset number reaches a reset threshold; if not, returning to the obtaining of the sample face image; if yes, ending the training to obtain the emotion recognition model.

6. A method of emotion recognition characterized by, comprises: obtaining a to-be-recognized face image; performing region extraction on the to-be-recognized face image to obtain a third region image containing an eye part in the to-be-recognized face image and a fourth region image containing a mouth part in the to-be-recognized face image; performing feature fusion on the third region image and the fourth region image to obtain a second fusion feature; inputting the second fusion feature into an emotion recognition model to obtain an emotion recognition result; the emotion recognition model is trained by the training method of the emotion recognition model according to any one of claims 1 to 5.

7. The emotion recognition method of claim 6, wherein, Before the performing of the feature fusion on the third region image and the fourth region image to obtain a second fusion feature, the method further comprises: inputting the third region image into a pre-trained glasses removal network to perform mask operation on a glasses feature and a glasses shadow feature to obtain a third region image after the glasses are removed; the glasses removal network is trained based on an adversarial learning mechanism.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method according to any one of claims 1 to 7.

Images