Emotion recognition method and device

Abstract

The embodiment of the invention provides an emotion recognition method and device, and the method comprises the steps: determining an initial sample group, and obtaining initial face data based on the initial sample group; performing data annotation on the face data, and determining annotated data; performing data preprocessing based on the annotation data, and determining target face data; performing feature extraction based on the target facial data, and determining expression features; training a deep learning model based on the expression features, and determining an emotion recognition model; and acquiring to-be-recognized face data through a camera, and determining emotion data from the to-be-recognized face data through the emotion recognition model. The method realizes automatic detection and recognition of micro-expressions by constructing a deep learning model, has high precision, high efficiency and wide application prospect, and can be applied to the fields of man-machine interaction, intelligent monitoring, mental health and the like.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of mental health technology, and in particular, to emotion recognition methods. Background Art

[0002] Emotion recognition is one of the important research directions in the field of artificial intelligence, and is of great significance for improving human-computer interaction and user experience. Micro-expressions are short-lived and rapid changes in facial expressions, usually only a fraction of a second, but can express rich emotional information. Due to the short-lived and unconscious nature of micro-expressions, traditional image processing and computer vision technologies are difficult to effectively recognize them. Therefore, how to accurately recognize micro-expressions has become an important problem in the field of emotion recognition.

[0003] Traditional emotion recognition methods mainly rely on features such as voice and facial expressions, but these features may be limited in some cases, such as voice disguise and occlusion. Therefore, finding a new emotion recognition method to improve the accuracy and robustness of emotion recognition is a hot topic in current research. Summary of the invention

[0004] In view of this, the embodiments of this specification provide an emotion recognition method. One or more embodiments of this specification also relate to an emotion recognition device, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an emotion recognition method, comprising: Determine an initial sample population, and obtain initial facial data based on the initial sample population; Annotate the facial data and determine the annotated data; Perform data preprocessing based on the labeled data to determine the target facial data; Extract features based on target facial data to determine expression features; Train the deep learning model based on facial expression features and determine the emotion recognition model; The facial data to be identified is obtained through the camera, and the facial data to be identified is passed through the emotion recognition model to determine the emotion data.

[0006] In a possible implementation, obtaining initial facial data based on an initial sample population includes: Determine multiple types of emotions, multiple lighting parameters, and multiple orientation parameters; Initial facial data is obtained based on multiple types of emotions, multiple lighting parameters, and multiple orientation parameters.

[0007] In a possible implementation, data preprocessing is performed based on the labeled data to determine the target facial data, including: For the denoising operation, the normalization operation, and the image enhancement operation, target facial data is determined.

[0008] In a possible implementation, feature extraction is performed based on target facial data to determine expression features, including: Construct a multi-layer convolutional neural network model, extract features of the target facial data based on the multi-layer convolutional neural network model, and determine the expression features.

[0009] In a possible implementation, a deep learning model is trained based on expression features to determine an emotion recognition model, including: Determine an initial deep learning model; wherein the initial deep learning model includes a classifier and a regressor; Determine the loss function, iteratively adjust the parameters of the initial deep learning model through the gradient descent algorithm and the loss function, and determine the emotion recognition model.

[0010] In a possible implementation, the method further includes: Regularization technology is used to constrain the emotion recognition model, and early stopping or Dropout technology is used to optimize the emotion recognition model.

[0011] In a possible implementation, the method further includes: Evaluate the emotion recognition model by a cross-validation method to determine performance data of the emotion recognition model; the performance data includes at least one of accuracy, recall rate and F1 value; When the performance data is less than the performance threshold, the emotion recognition model is optimized.

[0012] According to a second aspect of the embodiments of this specification, there is provided an emotion recognition device, comprising: An initial data determination module is configured to determine an initial sample population and obtain initial facial data based on the initial sample population; A labeling data determination module is configured to label the facial data and determine the labeling data; A data preprocessing module is configured to perform data preprocessing based on the labeled data to determine target facial data; A feature extraction module is configured to extract features based on the target facial data to determine expression features; A model training module is configured to train a deep learning model based on expression features to determine an emotion recognition model; The emotion determination module is configured to obtain the facial data to be identified through the camera, and determine the emotion data by passing the facial data to be identified through the emotion recognition model.

[0013] According to a third aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned emotion recognition method are implemented.

[0014] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned emotion recognition method are implemented.

[0015] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned emotion recognition method.

[0016] The embodiments of this specification provide an emotion recognition method and device, wherein the emotion recognition method includes: determining an initial sample population, obtaining initial facial data based on the initial sample population; annotating the facial data to determine the annotated data; performing data preprocessing based on the annotated data to determine the target facial data; extracting features based on the target facial data to determine the expression features; training a deep learning model based on the expression features to determine the emotion recognition model; obtaining facial data to be recognized through a camera, passing the facial data to be recognized through the emotion recognition model to determine the emotion data. Automatic detection and recognition of micro-expressions is achieved by constructing a deep learning model, which has high precision, high efficiency and broad application prospects, and can be applied to human-computer interaction, intelligent monitoring, mental health and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of an emotion recognition method provided by an embodiment of this specification; Figure 2 is a schematic diagram of an emotion recognition method provided by an embodiment of this specification; Figure 3 is a schematic diagram of the structure of an emotion recognition device provided by an embodiment of this specification; Figure 4 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0018] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0019] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0021] In this specification, an emotion recognition method is provided. This specification also relates to an emotion recognition apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0022] See also Figure 1 , Figure 1 A flowchart of an emotion recognition method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0023] Step 101: Determine an initial sample population, and obtain initial facial data based on the initial sample population.

[0024] In a possible implementation, initial facial data is obtained based on an initial sample population, including: determining multiple types of emotions, multiple lighting parameters, and multiple orientation parameters; and obtaining initial facial data based on the multiple types of emotions, multiple lighting parameters, and multiple orientation parameters.

[0025] In practical applications, see Figure 2 ,Collect facial image data containing micro-expressions of seven emotions (angry, disgusted, worried, happy, sad, surprised, and natural), including a diverse data set of frontal, side, different lighting conditions, and different expression states.

[0026] Step 102: Annotate the facial data to determine the annotated data.

[0027] In practical applications, after obtaining sample data, the data can be labeled, such as seven emotions: anger, disgust, worry, happiness, sadness, surprise, and natural.

[0028] Step 103: Perform data preprocessing based on the labeled data to determine target facial data.

[0029] In a possible implementation, data preprocessing is performed based on the labeled data to determine the target facial data, including: performing a denoising operation, a normalization operation, and an image enhancement operation to determine the target facial data.

[0030] In practical applications, the collected image data is preprocessed, including denoising, normalization, enhancement, etc. Median filtering is used to remove noise in the image; histogram equalization is used to enhance image contrast; affine transformation and other methods are used to normalize the image.

[0031] Furthermore, in order to improve the generalization ability of the model, data enhancement methods can be used to expand the training data. For example, random cropping, rotation, translation and other operations can be used to transform the image to generate more training samples.

[0032] Step 104: Extract features based on the target facial data to determine expression features.

[0033] In a possible implementation, feature extraction is performed based on target facial data to determine expression features, including: constructing a multi-layer convolutional neural network model, and extracting features from the target facial data based on the multi-layer convolutional neural network model to determine expression features.

[0034] In practical applications, deep learning algorithms such as convolutional neural networks (CNNs) are used to extract features from preprocessed image data. Specifically, a multi-layer convolutional neural network model can be constructed to extract features related to micro-expressions through operations such as convolutional layers and pooling layers. At the same time, some auxiliary technologies such as data enhancement can be used to improve the generalization ability of the model.

[0035] Step 105: Train a deep learning model based on facial expression features to determine an emotion recognition model.

[0036] In one possible implementation, a deep learning model is trained based on facial expression features to determine an emotion recognition model, including: determining an initial deep learning model; wherein the initial deep learning model includes a classifier and a regressor; determining a loss function, and iteratively adjusting the parameters of the initial deep learning model through a gradient descent algorithm and the loss function to determine the emotion recognition model.

[0037] In practical applications, the extracted features are used to train deep learning models, including classifiers, regressors, etc. Specifically, a fully connected layer or support vector machine classifier is constructed to classify micro-expressions. At the same time, the gradient descent algorithm is used to optimize the model.

[0038] In a possible implementation, it also includes: using regularization technology to constrain the emotion recognition model, and using early stopping method or Dropout technology to optimize the emotion recognition model.

[0039] In practical applications, some optimization strategies are used to improve problems that may occur during model training, such as overfitting and underfitting. For example, regularization techniques can be used to constrain the model to avoid overfitting; early stopping and Dropout techniques can be used to prevent underfitting.

[0040] Furthermore, for application scenarios with high real-time requirements, some optimization strategies can be used to improve the running speed of the model. For example, lightweight neural network models, hardware acceleration and other technologies can be used to improve the running efficiency of the model.

[0041] In a possible implementation, it also includes: evaluating the emotion recognition model through a cross-validation method to determine the performance data of the emotion recognition model; the performance data includes at least one of accuracy, recall rate and F1 value; when the performance data is less than the performance threshold, optimizing the emotion recognition model.

[0042] In practical applications, the trained model is evaluated using test data. The performance of the model can be measured using indicators such as accuracy, recall, and F1 value. At the same time, methods such as cross-validation can be used to improve the accuracy of the evaluation.

[0043] Step 106: Acquire the facial data to be identified through the camera, and determine the emotion data by passing the facial data to be identified through the emotion recognition model.

[0044] During the application process, the user's privacy information needs to be protected. Some privacy protection technologies, such as differential privacy and homomorphic encryption, can be used to desensitize user data to ensure the security of user data.

[0045] In practical applications, when this solution is applied in fields such as psychological counseling, after the set emotion is identified, an early warning of the emotion can be issued, and certain suggestions can be given to improve the application effect.

[0046] The embodiments of this specification provide an emotion recognition method and device, wherein the emotion recognition method includes: determining an initial sample population, obtaining initial facial data based on the initial sample population; annotating the facial data to determine the annotated data; performing data preprocessing based on the annotated data to determine the target facial data; extracting features based on the target facial data to determine the expression features; training a deep learning model based on the expression features to determine the emotion recognition model; obtaining facial data to be recognized through a camera, passing the facial data to be recognized through the emotion recognition model to determine the emotion data. Automatic detection and recognition of micro-expressions is achieved by constructing a deep learning model, which has high precision, high efficiency and broad application prospects, and can be applied to human-computer interaction, intelligent monitoring, mental health and other fields.

[0047] In summary, the automatic detection and recognition of micro-expressions can be achieved by building a deep learning model. This technology has high precision, high efficiency and broad application prospects, and can be applied to human-computer interaction, intelligent monitoring, mental health and other fields. At the same time, this technology can be further optimized and improved, such as using model optimization, data enhancement, multimodal fusion and other technologies to improve recognition accuracy and robustness; using real-time optimization technology to improve operation efficiency; using privacy protection technology to protect user privacy; using explainability enhancement technology to improve the transparency and explainability of the model.

[0048] Corresponding to the above method embodiment, this specification also provides an emotion recognition device embodiment, Figure 3 FIG. 1 is a schematic diagram showing the structure of an emotion recognition device provided by an embodiment of the present specification. Figure 3 As shown, the device comprises: The initial data determination module 301 is configured to determine an initial sample population and obtain initial facial data based on the initial sample population; The labeling data determining module 302 is configured to label the facial data and determine the labeling data; The data preprocessing module 303 is configured to perform data preprocessing based on the labeled data to determine the target facial data; A feature extraction module 304 is configured to perform feature extraction based on the target facial data to determine expression features; A model training module 305 is configured to train a deep learning model based on expression features to determine an emotion recognition model; The emotion determination module 306 is configured to obtain the facial data to be identified through the camera, and determine the emotion data by applying the facial data to be identified to the emotion recognition model.

[0049] In a possible implementation, obtaining initial facial data based on an initial sample population includes: Determine multiple types of emotions, multiple lighting parameters, and multiple orientation parameters; Initial facial data is obtained based on multiple types of emotions, multiple lighting parameters, and multiple orientation parameters.

[0050] In a possible implementation, data preprocessing is performed based on the labeled data to determine the target facial data, including: For the denoising operation, the normalization operation, and the image enhancement operation, target facial data is determined.

[0051] In a possible implementation, feature extraction is performed based on target facial data to determine expression features, including: Construct a multi-layer convolutional neural network model, extract features of the target facial data based on the multi-layer convolutional neural network model, and determine the expression features.

[0052] In a possible implementation, a deep learning model is trained based on expression features to determine an emotion recognition model, including: Determine an initial deep learning model; wherein the initial deep learning model includes a classifier and a regressor; Determine the loss function, iteratively adjust the parameters of the initial deep learning model through the gradient descent algorithm and the loss function, and determine the emotion recognition model.

[0053] In a possible implementation, the method further includes: Regularization technology is used to constrain the emotion recognition model, and early stopping or Dropout technology is used to optimize the emotion recognition model.

[0054] In a possible implementation, the method further includes: Evaluate the emotion recognition model by a cross-validation method to determine performance data of the emotion recognition model; the performance data includes at least one of accuracy, recall rate and F1 value; When the performance data is less than the performance threshold, the emotion recognition model is optimized.

[0055] The embodiments of this specification provide an emotion recognition method and device, wherein the emotion recognition device comprises: determining an initial sample population, obtaining initial facial data based on the initial sample population; annotating the facial data to determine the annotated data; performing data preprocessing based on the annotated data to determine the target facial data; extracting features based on the target facial data to determine the expression features; training a deep learning model based on the expression features to determine the emotion recognition model; obtaining facial data to be recognized through a camera, passing the facial data to be recognized through the emotion recognition model to determine the emotion data. Automatic detection and recognition of micro-expressions is achieved by constructing a deep learning model, which has high precision, high efficiency and broad application prospects, and can be applied to human-computer interaction, intelligent monitoring, mental health and other fields.

[0056] The above is a schematic scheme of an emotion recognition device of this embodiment. It should be noted that the technical scheme of the emotion recognition device and the technical scheme of the above emotion recognition method belong to the same concept, and the details not described in detail in the technical scheme of the emotion recognition device can be referred to the description of the technical scheme of the above emotion recognition method.

[0057] Figure 4 The block diagram of a computing device 400 according to one embodiment of the present specification is shown. The components of the computing device 400 include but are not limited to a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and the database 450 is used to store data.

[0058] The computing device 400 also includes an access device 440 that enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0059] In one embodiment of the present specification, the above components of the computing device 400 and Figure 4 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 4 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0060] The computing device 400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 400 may also be a mobile or stationary server.

[0061] The processor 420 is used to execute the following computer executable instructions, which implement the steps of the above-mentioned emotion recognition method when executed by the processor. The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned emotion recognition method belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be found in the description of the technical scheme of the above-mentioned emotion recognition method.

[0062] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned emotion recognition method.

[0063] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned emotion recognition method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned emotion recognition method.

[0064] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned emotion recognition method.

[0065] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above emotion recognition method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above emotion recognition method.

[0066] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0067] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0068] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0069] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An emotion recognition method, characterized in that: include: Determining an initial sample population, and acquiring initial facial data based on the initial sample population; Performing data labeling on the facial data to determine the labeling data; Performing data preprocessing based on the labeled data to determine target facial data; Perform feature extraction based on the target facial data to determine expression features; Based on the facial expression features, a deep learning model is trained to determine an emotion recognition model; The facial data to be identified is acquired through a camera, and the facial data to be identified is passed through the emotion recognition model to determine the emotion data.

2. The method according to claim 1, characterized in that The obtaining of initial facial data based on the initial sample population includes: Determine multiple types of emotions, multiple lighting parameters, and multiple orientation parameters; Initial facial data is acquired based on the multiple categories of emotions, the multiple lighting parameters, and the multiple orientation parameters.

3. The method according to claim 1, characterized in that The performing data preprocessing based on the labeled data to determine the target facial data includes: For the denoising operation, the normalization operation and the image enhancement operation, target facial data is determined.

4. The method according to claim 1, characterized in that: The step of extracting features based on the target facial data to determine expression features includes: A multi-layer convolutional neural network model is constructed, and feature extraction is performed on the target facial data based on the multi-layer convolutional neural network model to determine expression features.

5. The method according to claim 1, characterized in that The step of training a deep learning model based on the facial expression features to determine an emotion recognition model includes: Determine an initial deep learning model; wherein the initial deep learning model includes a classifier and a regressor; A loss function is determined, and the parameters of the initial deep learning model are iteratively adjusted through a gradient descent algorithm and the loss function to determine an emotion recognition model.

6. The method according to claim 5, characterized in that Also includes: Regularization technology is used to constrain the emotion recognition model, and early stopping method or Dropout technology is used to optimize the emotion recognition model.

7. The method according to claim 1, characterized in that Also includes: Evaluating the emotion recognition model by a cross-validation method to determine performance data of the emotion recognition model; The performance data includes at least one of accuracy, recall and F1 value; When the performance data is less than a performance threshold, the emotion recognition model is optimized.

8. An emotion recognition device, characterized in that: include: An initial data determination module is configured to determine an initial sample population and obtain initial facial data based on the initial sample population; a labeling data determining module, configured to label the facial data and determine the labeling data; A data preprocessing module is configured to perform data preprocessing based on the labeled data to determine target facial data; A feature extraction module is configured to perform feature extraction based on the target facial data to determine expression features; A model training module is configured to train a deep learning model based on the expression features to determine an emotion recognition model; The emotion determination module is configured to obtain the facial data to be identified through a camera, and determine the emotion data by passing the facial data to be identified through the emotion recognition model.

9. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the emotion recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the emotion recognition method according to any one of claims 1 to 7.

Images