Emotion evaluation method and system

Abstract

The invention belongs to the technical field of emotion evaluation. The emotion evaluation method comprises the steps that a video frame of a played video in a preset time period and an expression frame of micro-expression change of a user when the user watches the video are obtained, and the expression frame represents the facial expression of the user recorded by a frame of image; performing corresponding matching on the video frame and the expression frame to obtain to-be-analyzed data; inputting the to-be-analyzed data into a target convolutional neural network to obtain a micro-expression data analysis result; and obtaining an emotion evaluation result of the user according to the micro-expression data analysis result. Through the method, the related characteristics of the expression change are analyzed to summarize the emotional state of the testee when the testee receives the test, and a reasonable reference analysis material is provided for related services of psychological assessment.

Description

Technical Field

[0001] The present application belongs to the technical field of emotion assessment, and in particular, relates to an emotion assessment method and system. Background Art

[0002] With the development of emotion assessment technology, the demand for accurately understanding and quantifying personal emotional states is growing. Emotion assessment not only has profound academic value in the fields of psychology and neuroscience, but also shows broad application prospects in many social practice fields such as business, education, and medical care.

[0003] Traditional emotion assessment methods mainly rely on psychological methods such as questionnaires, interviews and observations. Although these methods can provide information about emotional states to a certain extent, they have limitations such as being time-consuming, inefficient and highly subjective. Summary of the invention

[0004] Based on this, it is necessary to provide an emotion assessment method and system to solve the above technical problems.

[0005] In a first aspect, the present application provides a method for emotion assessment, the method comprising:

[0006] Obtain video frames of the video played within a preset time period, and expression frames of the user's micro-expression changes when watching the video, wherein the expression frame represents the user's facial expression recorded by a frame of image;

[0007] Matching the video frame with the expression frame to obtain data to be analyzed;

[0008] Inputting the data to be analyzed into the target convolutional neural network to obtain micro-expression data analysis results;

[0009] Based on the micro-expression data analysis results, the user's emotion evaluation results are obtained.

[0010] In some practicable methods, the step of obtaining video frames of a video played within a preset time period, and expression frames of a user's micro-expression changes while watching the video, includes:

[0011] Acquire a pre-built video to obtain a video frame, wherein the video at least includes a strong emotional stimulation element that can cause a change in the user's mentality;

[0012] According to the time when the video is played, the micro-expression changes of the user's face when watching the video are obtained to obtain expression frames.

[0013] In some practicable manners, the step of matching the video frame with the expression frame to obtain the data to be analyzed includes:

[0014] Assigning a timestamp to each of the video frames and each of the expression frames to obtain the video frames and the expression frames having the timestamp;

[0015] According to the video frame and the expression frame having the timestamp, corresponding matching is performed between the video frame and the expression frame to obtain data to be analyzed.

[0016] In some practicable methods, the step of inputting the data to be analyzed into a target convolutional neural network to obtain micro-expression data analysis results includes:

[0017] Performing alignment and clipping preprocessing on the data to be analyzed to obtain target analysis data;

[0018] Analyzing the target data, performing feature extraction, and obtaining micro-expression feature information;

[0019] The micro-expression feature information is input into the target convolutional neural network to obtain micro-expression data analysis results.

[0020] In some practicable methods, the step of analyzing the target data, extracting features, and obtaining micro-expression feature information includes:

[0021] Performing optical flow motion intensity analysis on the expression frames in the target analysis data to determine the peak frame with the highest intensity of each micro-expression;

[0022] Taking the peak frame as the center, optical flow field calculation is performed with a preset number of preceding and following frames to obtain optical flow field information in the preceding and following frames of the peak frame, wherein the preceding and following frames represent a preset number of frames before the peak frame and a preset number of frames after the peak frame;

[0023] The micro-expression feature information is obtained according to the optical flow field information.

[0024] In some practicable methods, the step of inputting the micro-expression feature information into a target convolutional neural network to obtain a micro-expression data analysis result includes:

[0025] Construct a convolutional neural network model with a 3D CNN architecture and obtain an initial convolutional neural network;

[0026] Using historical micro-expression data to train an initial convolutional neural network to obtain the target convolutional neural network;

[0027] The micro-expression feature information is input into the target convolutional neural network to obtain micro-expression data analysis results.

[0028] In some practicable methods, the step of obtaining the user's emotion evaluation result based on the micro-expression data analysis result includes:

[0029] According to the micro-expression data analysis results, the probability of predicting micro-expressions is obtained;

[0030] According to the probability of predicting micro-expressions, the user's emotion evaluation result is obtained.

[0031] In a second aspect, the present application provides an emotion assessment system, which is applied to the above-mentioned emotion assessment method, and the system comprises:

[0032] An acquisition unit, used to acquire video frames of a video played within a preset time period, and expression frames of a user's micro-expression changes when watching a video, wherein the expression frame represents a user's facial expression recorded by a frame of image;

[0033] A matching unit, used for matching the video frame with the expression frame to obtain data to be analyzed;

[0034] An analysis unit, used to input the data to be analyzed into a target convolutional neural network to obtain micro-expression data analysis results;

[0035] The result unit is used to obtain the user's emotion evaluation result based on the micro-expression data analysis result.

[0036] In a third aspect, the present application provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the aforementioned emotion assessment method are implemented.

[0037] In a fourth aspect, the present application provides a computer program, which, when executed by a processor, implements the steps of the aforementioned emotion assessment method.

[0038] Beneficial effects: A method for evaluating emotions, obtaining video frames of a video played within a preset time period, and expression frames of micro-expression changes of a user while watching a video, wherein the expression frame represents a facial expression of a user recorded in a frame of image; matching the video frame with the expression frame to obtain data to be analyzed; inputting the data to be analyzed into a target convolutional neural network to obtain micro-expression data analysis results; and obtaining the emotion evaluation results of the user based on the micro-expression data analysis results. Through the above method, the relevant features of expression changes are analyzed to summarize the emotional state of the person being tested when taking the test, providing reasonable reference analysis materials for psychological assessment related businesses. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flowchart of a method for evaluating emotions in one embodiment;

[0041] Figure 2 is a diagram illustrating an example of a method for evaluating emotions in one embodiment;

[0042] Figure 3 The figure is a logic diagram of a method for emotion assessment in one embodiment. DETAILED DESCRIPTION

[0043] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all couplings of one or more related listed items.

[0045] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0046] Some terms involved in this application are explained below to facilitate understanding of this application:

[0047] The Network Time Protocol (NTP) is a network protocol used for computer clock synchronization. It allows computer clients and servers to accurately synchronize their clocks to a unified time standard.

[0048] Haar Cascades is a machine learning algorithm for object detection, suitable for real-time visual object detection tasks such as face, eye, and even vehicle detection. Haar Cascades trains classifiers by using Haar features (a simple and effective image feature) and AdaBoost algorithm (a method to improve the performance of weak classifiers). These classifiers are able to identify specific objects in images.

[0049] HOG (Histogram of Oriented Gradients) is a feature descriptor used for image recognition and object detection. It captures shape information by calculating the histogram of gradient orientation in a local area of ​​an image.

[0050] SVM (Support Vector Machine) is a supervised learning model used for classification and regression analysis. HOG+SVM combines HOG feature extraction and SVM classifier to improve the accuracy of object detection. In tasks such as pedestrian detection, HOG features are extracted and then classified using SVM to identify pedestrians in the image.

[0051] MTCNN (Multi-Task Cascaded Convolutional Networks) is a deep learning framework for face detection. MTCNN consists of three stages of neural networks, each of which performs a different task: the first stage is a rough detection of face candidate regions, the second stage is accurate facial feature point localization, and the third stage is further refinement of the face region.

[0052] The Active Shape Model (ASM) is a statistical model used to locate feature points in images. It is based on the Point Distribution Model (PDM) and obtains statistical information on the distribution of feature points through training image samples. It also determines the direction in which feature points are allowed to change, so as to find the corresponding feature point positions on the target image.

[0053] Spatiotemporal features refer to features in two dimensions: time and space. They describe the spatial distribution and temporal variation of data.

[0054] A 3D convolutional neural network is a deep learning architecture that extends the traditional 2D CNN to the third dimension, the time dimension. This network is able to process inputs with volumetric data (such as video frame sequences) and extract features by performing convolution operations simultaneously in space and time.

[0055] The Savitzky-Golay smoothing filter algorithm is a digital filtering technique widely used in signal processing and data analysis.

[0056] The Farneback optical flow algorithm is a dense optical flow algorithm that approximates the motion field through polynomial expansion and uses phase difference to estimate the motion vector.

[0057] like Figure 1 As shown, in the first aspect, the present application provides a method for emotion assessment, the method comprising:

[0058] S100, obtaining video frames of a video played within a preset time period, and expression frames of micro-expression changes of a user while watching the video.

[0059] The expression frame represents a frame of image recording the facial expression of the user.

[0060] The purpose of acquiring video frames and expression frames is to collect video content (video frames) played within a specific time period and the changes in user expressions when watching these video contents (expression frames) for subsequent emotion analysis.

[0061] Specifically, obtaining video frames of a video played within a preset time period, and expression frames of a user's expression changes when watching a video, may include the following steps:

[0062] S101, obtaining a pre-built video to obtain a video frame.

[0063] The video at least includes strong emotional stimulation elements that can cause changes in the user's mentality.

[0064] Specifically, the constructed videos are not regular videos, but are designed to trigger or mobilize specific emotions in users. In other words, these videos contain strong emotional stimulation elements, such as tragedy, comedy, horror, surprise, etc., to ensure that they can cause changes in the user's mentality. Video frame acquisition refers to capturing each frame of the video at a fixed time interval (for example, 24 or 30 frames per second) through a video recording device or software during video playback. These frames will serve as the basis for analysis for subsequent emotion recognition.

[0065] For example, the video contains strong emotional stimulation elements:

[0066] 1. Emotionally rich film clips: Film clips containing elements of tragedy, comedy, horror, suspense, etc., which can trigger emotions such as anger, sadness, fear, happiness, and astonishment in the audience.

[0067] 2. News Reports: News videos reporting major events, disasters, and touching stories can evoke strong emotional responses from viewers.

[0068] 3. Documentary: Documentaries that showcase human challenges, natural wonders, social issues, etc. can inspire a variety of emotions in the audience.

[0069] 4. Music Videos: Music videos can evoke emotional resonance in the audience through the combination of music and visual elements. 5. Emotional Interviews: Personal story sharing in interview programs, especially those involving personal challenges, triumphs or losses, can evoke sympathy and resonance in the audience.

[0070] 6. Public Service Announcements: Public service announcements aimed at raising public awareness, such as about environmental protection, child welfare, animal rights, etc., can arouse the audience's sympathy and anger.

[0071] When selecting video content, the target audience and the purpose of the assessment should be considered to ensure that the video content can both stimulate emotional responses and meet ethical and legal standards. In emotional assessment, the selection of video content should match the assessment objectives to ensure accurate and valid emotional data.

[0072] S102, obtaining, according to the time of playing the video, changes in the user's facial micro-expressions when watching the video, and obtaining expression frames.

[0073] Specifically, expression capture refers to the use of a high-speed camera to record the changes in the viewer's facial expressions in real time while the video is playing. The use of a high-speed camera can ensure that there is enough resolution and frame rate to capture subtle changes in expression. Expression frame synchronization is used to ensure that the capture of expression frames is synchronized with the capture of video frames. This is usually achieved by using the same timestamp or synchronization signal between the video playback and the expression capture device.

[0074] S200, matching the video frame with the expression frame to obtain data to be analyzed.

[0075] The purpose of matching the video frames with the expression frames is to ensure that the video content and the viewer's (user's) micro-expression reactions precisely correspond in time, so as to perform accurate emotion analysis.

[0076] Specifically, obtaining the data to be analyzed may include the following steps:

[0077] S201, assigning a timestamp to each of the video frames and each of the expression frames to obtain the video frames and the expression frames having the timestamp.

[0078] During the video playback and expression capture process, an accurate timestamp is assigned to each frame of video and each frame of expression. This timestamp reflects the exact time point when the frame was captured. To ensure the time synchronization of the video playback device and the expression capture device, a common clock signal or network time protocol (NTP) can be used to achieve this.

[0079] S202 , performing corresponding matching between the video frame and the expression frame according to the video frame and the expression frame having the timestamp, to obtain data to be analyzed.

[0080] The same timestamp is used in this application for ease of understanding. In fact, there is a time difference between a user watching a video and forming a corresponding micro-expression, which is generally between 1 / 25 second and 1 / 5 second. Therefore, the video frame needs to be synchronized with the micro-expression delayed between 1 / 25 second and 1 / 5 second. In this way, the synchronization of the video frame and the expression frame is expressed as the following formula:

[0081] V t The timestamp of the video frame.

[0082] E t The timestamp of the expression frame.

[0083] Δt is the time difference between the video frame and the expression frame. Seconds to Between seconds.

[0084] The time difference Δt can be expressed as formula (1):

[0085]

[0086] In order to ensure the synchronization of video frames and expression frames, the following relationship can be established:

[0087] E t =V t +Δt (2)

[0088] This formula (2) shows that the timestamp E of the expression frame t Equal to the timestamp V of the video frame t Add the time difference Δt.

[0089] In this way, the precise temporal correspondence between video frames and expression frames can be ensured, thus providing an accurate data basis for emotion analysis.

[0090] S300, inputting the data to be analyzed into a target convolutional neural network to obtain micro-expression data analysis results.

[0091] Specifically, obtaining micro-expression data analysis results may include the following steps:

[0092] S301, performing alignment and clipping preprocessing on the data to be analyzed to obtain target analysis data.

[0093] S302, performing feature extraction on the target analysis data to obtain micro-expression feature information.

[0094] S303, inputting the micro-expression feature information into a target convolutional neural network to obtain micro-expression data analysis results.

[0095] It should be noted that the purpose of obtaining micro-expression data is to analyze micro-expression data and extract information about emotional states through convolutional neural networks (CNN). In step S301, a face detection algorithm (such as HaarCascades, HOG+SVM, MTCNN, etc.) is used to locate the center point of the eye area of ​​the face in the video captured by the camera (the video captured by the camera) to determine the initial position of the face; the active shape model (ASM) is used to fit the exact position of the face shape and crop the face area; the local weighted average (LWM) transformation is used to align all facial areas in the sequence frame by frame with the area in the reference frame. For example, the first frame of the expression frame can be used as the reference frame to ensure that the position of the face in the expression frame is consistent, which is convenient for subsequent processing. Next, the detected face area is cropped from the original expression frame and adjusted to a uniform size. Then, in step S302, micro-expression features are extracted from the facial expression sequence, and redundancy between features is removed to reduce feature dimensions. This can be achieved by using different feature extraction techniques, such as local binary pattern (LBP), edge orientation histogram (EOH), etc. Finally, the micro-expression feature information is input into the target convolutional neural network model, which processes the input features and outputs the predicted probability or regression score of each category (such as anger, happiness, etc.), and the predicted probability or regression score is used as the micro-expression data analysis result.

[0096] It should also be noted that, S302, extracting features from the target analysis data to obtain micro-expression feature information, may include the following steps:

[0097] S3021, performing optical flow motion intensity analysis on the expression frames in the target analysis data to determine the peak frame with the highest intensity of each micro-expression.

[0098] Specifically, first, the optical flow algorithm is used to calculate the optical flow between consecutive frames in the video to capture the dynamic changes of facial micro-expressions. Next, by analyzing the intensity of the optical flow motion, the peak frame with the highest micro-expression intensity is found, that is, the moment when the micro-expression changes most dramatically.

[0099] It should be noted that in a 1-minute video, there may be multiple expression events formed by strong emotional stimulation elements. Therefore, the user's expression will also have multiple peak frames with the highest micro-expression intensity within this 1 minute, that is, the moment when the micro-expression changes most dramatically. Therefore, it is necessary to identify and process multiple micro-expression events (the processing method of each micro-expression event is the same, and each micro-expression event will not be elaborated). Therefore, the key areas of the user's face are selected as the areas of interest, such as eyebrows, corners of the eyes, corners of the mouth, etc. These areas are most active in micro-expression changes.

[0100] Next, the Farneback optical flow method is used to calculate the dense optical flow in the region of interest to obtain the changes in micro-expressions. By calculating the change in the amplitude of the optical flow vector in each region of interest, the amplitude curve of the optical flow motion intensity over time is obtained. This curve reflects the dynamic changes of micro-expressions over time.

[0101] Then, the Savitzky-Golay smoothing filter algorithm is used to filter the amplitude change curve to eliminate noise and highlight the significant points of micro-expression changes. These significant points correspond to the most drastic moments of micro-expression changes, and each significant point will be used as a peak frame of a micro-expression event. By fitting a local polynomial to smooth the data, that is, filtering the amplitude change curve, the shape and characteristics of the micro-expression can be retained at the same time.

[0102] It should also be noted that, since each micro-expression event is processed in the same way, the processing method of an expression event is used as an example. In the aforementioned step S3021, a smoothed amplitude change curve has been obtained. Next, the significant point (peak frame) is analyzed, and the optical flow changes of the frames before and after the peak frame are analyzed to determine the start and end of the expression event. When a micro-expression event occurs, it is usually manifested as a sudden increase in optical flow changes, followed by a gradual decrease. By analyzing the pattern of optical flow changes (the pattern refers to the shape or structure of the optical flow changes, that is, the direction and order of movement, which describes the distribution characteristics of the optical flow changes over time, and the pattern can reflect the dynamic characteristics of specific micro-expression changes. For example, a smiling micro-expression event may be manifested as a movement pattern of the corners of the mouth upward, while a surprised micro-expression event may be manifested as a pattern of eyebrows raised and eyes wide open.) and amplitude (amplitude refers to the intensity or size of the optical flow change, that is, the intensity of movement, which quantifies the degree of pixel movement in a specific area. In micro-expression analysis, the amplitude can reflect the degree of movement of specific facial areas (such as eyebrows, corners of the eyes, and corners of the mouth), thereby indicating the intensity of micro-expressions), different micro-expression events are distinguished. The beginning of each micro-expression event is usually accompanied by a significant increase in optical flow change, while the end is a decrease in optical flow change.

[0103] The purpose of analyzing the pattern and magnitude of optical flow changes is to:

[0104] Patterns are crucial for distinguishing different types of micro-expressions, as different micro-expressions may involve movements of different parts of the face, and even if the amplitudes are similar, the patterns may be completely different.

[0105] Amplitude can help identify the intensity of micro-expressions and is important for distinguishing small changes in expression.

[0106] In this application, mode and amplitude are used together to provide different perspectives on micro-expression changes:

[0107] 1. Distinguishing expression types: By analyzing the patterns of optical flow, specific micro-expression types such as smile, surprise or anger can be identified.

[0108] 2. Evaluating the intensity of micro-expressions: By analyzing the amplitude of optical flow, the intensity of micro-expressions can be evaluated, because micro-expressions usually involve very subtle facial movements.

[0109] S3022, taking the peak frame as the center, and performing optical flow calculation on a preset number of frames before and after the peak frame to obtain optical flow information in the frames before and after the peak frame.

[0110] The preceding and following frames represent a preset number of frames before the peak frame and a preset number of frames after the peak frame.

[0111] Specifically, n frames before the peak frame and m frames after the peak frame can be selected to form a time window, where n and m can be the same or different. When n and m are different, m is greater than n, and this window contains the key moments of micro-expression changes. For each frame in this time window, the optical flow algorithm is used to calculate the optical flow field between it and the peak frame, providing the motion information of each pixel in the horizontal and vertical directions. The calculated optical flow information is integrated to form the optical flow field information of the peak frame and its previous and subsequent frames. The optical flow field information describes the motion pattern of the key facial area during the micro-expression change process (the motion pattern is not just a single motion feature, but a set of features, including the direction, speed, acceleration, etc. of the motion. These features together describe the dynamic behavior of the facial area in the expression change).

[0112] For example, when analyzing a 1-minute video, the video contains multiple emotional stimulation elements, which causes the user to show multiple micro-expressions.

[0113] 1. Determine the peak frame: Through step S3021, the peak frames of three different micro-expressions are determined.

[0114] 2. Select the previous and next frames: For each peak frame, select the first 5 frames and the last 5 frames to form a time window containing 11 frames.

[0115] 3. Optical flow field calculation: Use the optical flow method to calculate the optical flow field between the frame in each window and the peak frame.

[0116] S3023: Obtain the micro-expression feature information according to the optical flow field information.

[0117] Specifically, the optical flow field information is used as micro-expression feature information so as to be input into the target convolutional neural network in the subsequent steps.

[0118] It should be noted that, S3023, obtaining the micro-expression feature information according to the optical flow field information may include the following steps:

[0119] S30231, construct a convolutional neural network model of the 3D CNN architecture and obtain an initial convolutional neural network.

[0120] Specifically, a three-dimensional convolutional neural network (3D CNN) model is constructed to analyze micro-expression feature information. The initial convolutional neural network is able to capture both spatial and temporal features in the video. The construction process begins with defining the network architecture, including multiple 3D convolutional layers that can extract dynamic features from optical flow field information. Each 3D convolutional layer is followed by an activation function, such as ReLU, to introduce nonlinearity, allowing the network to learn complex feature representations. In addition, 3D pooling layers can be included to reduce the spatial dimension of the features, as well as normalization layers to speed up the training process and improve the generalization ability of the model. After the construction is completed, an initial convolutional neural network model is obtained, which will serve as the basis for training.

[0121] S30232, using historical micro-expression data to train the initial convolutional neural network to obtain the target convolutional neural network.

[0122] Specifically, it is trained using historical micro-expression data. These data may include expression frames collected in the previous step. When there are multiple expression events in a 1-minute video, each expression event corresponds to a label, which indicates the type of expression in the video. In other words, the micro-expression label represented by the expression frame is given, and the one-minute video played to the subject is an external means to stimulate the subject to produce micro-expression changes. The training process involves inputting these data into the initial convolutional neural network and adjusting the weights of the initial convolutional neural network to minimize the difference between the predicted and actual labels. This can be achieved through the back-propagation algorithm and the gradient descent method. As the training proceeds, the initial convolutional neural network gradually learns the features that distinguish different micro-expression categories. After the training is completed, the initial convolutional neural network model is transformed into a trained target convolutional neural network that can recognize micro-expressions, and it has the ability to predict new data.

[0123] S30233, input the micro-expression feature information into the target convolutional neural network to obtain micro-expression data analysis results.

[0124] Specifically, after the target convolutional neural network is trained, it will be used for the actual micro-expression analysis task. That is, the optical flow field information obtained in the previous steps is used as micro-expression feature information and input into the trained target convolutional neural network. The target convolutional neural network will process these input data and output the predicted probability of each micro-expression category. These outputs can be used to determine the type of expression in the expression frame. For example, the target convolutional neural network may predict that a certain frame is most likely to express anger, happiness or other emotions. Ultimately, these micro-expression data analysis results can be used for a variety of applications such as emotion recognition and mental health assessment, providing a powerful tool for understanding the emotional state of an individual.

[0125] It should be noted that the training data of the target convolutional neural network includes patterns and amplitudes, so that during the training process, the meanings of patterns and amplitudes in different situations can be comprehensively judged, and then the obtained meanings are matched with the pre-labeled training data to train the initial convolutional neural network. The target convolutional neural network can obtain patterns and amplitudes from the micro-expression feature information, and then use the output of the target convolutional neural network to form the micro-expression data analysis results.

[0126] For example, a subject's micro-expressions are being analyzed while watching a horror movie clip. Through the above steps, the optical flow field of the subject's expression is calculated, and the corresponding feature information is extracted. The feature information is formed into feature vectors, and these feature vectors are input into the trained target convolutional neural network. The target convolutional neural network outputs the following prediction probabilities: [0.2, 0.1, 0.7], corresponding to anger, happiness, and surprise, respectively.

[0127] S400, obtaining a user's emotion evaluation result based on the micro-expression data analysis result.

[0128] Specifically, obtaining the user emotion evaluation result may include the following steps:

[0129] S401, obtaining the probability of predicting micro-expressions according to the micro-expression data analysis results.

[0130] S402, obtaining an emotion evaluation result of the user according to the probability of predicting the micro-expression.

[0131] Specifically, when there is one or more expression events in a one-minute video, after watching, the user analyzes the expression frames formed, obtains the analysis results for each expression event, and then combines the analysis results of multiple events to finally obtain a comprehensive numerical score, that is, the user's emotion evaluation result. After obtaining the micro-expression with the highest prediction probability, this step will associate the micro-expression with the user's emotional state to obtain the emotion evaluation result.

[0132] For example, if there are multiple expression events in a one-minute video, the qualitative expressions such as "anger", "sadness", "fear", "happiness" and "surprise" in each expression event are plotted to form a chart. When forming the chart, the obtained probability can be converted into units for presentation on the chart.

[0133] When a 1-minute video contains multiple emotional stimulation elements, users will show multiple micro-expressions.

[0134] 1. Input the feature vector of an expression event into the trained target convolutional neural network, and the target convolutional neural network outputs the predicted probability of each micro-expression category.

[0135] 2. Determine the main emotion: Assume that the probability distribution of the model output is [0.1, 0.2, 0.7], corresponding to anger, happiness, and surprise respectively.

[0136] like Figure 2 As shown, 3. Emotion evaluation results: Based on the micro-expression with the highest predicted probability, it is concluded that the user mainly expresses surprise when watching the video. If there are multiple expression events in the video, repeat the above steps for each event, and combine the results by drawing a curve to obtain a comprehensive emotion score or emotional state description.

[0137] Example

[0138] like Figure 3 As shown, in this application, an emotion assessment method is used. The terminal device (psychological assessment terminal device) collects human faces in real time through a high-speed camera on it. The user needs to remain calm in front of the camera for one minute. During this minute, a video will be played on the terminal device screen, and the collected video screen will be transmitted to the back-end service through the network. The back-end service program will decompose the collected face video stream into expression frames, and the obtained expression frame images will be transmitted to the expression recognition analysis service, which will analyze the faces in the single-frame video images and analyze the proportion of various emotions in the expressions.

[0139] To ensure performance, the backend service program will asynchronously request the expression recognition service to analyze the emotion analysis results of the face in the video frame image. The results will be returned to the backend service program, and the emotional changes of the subject (user) in one minute will be obtained according to the time series of the expression frames.

[0140] Among them, the core of the expression recognition and analysis service lies in the CNN convolutional neural network, which classifies the emotions of pictures carried in the backend service requests. The classification may include "anger", "sadness", "fear", "happiness", "surprise", "depression", "other", and "contempt".

[0141] In addition, in order to provide API services to the outside world, the flask framework of Python can be used to encapsulate the target convolutional neural network.

[0142] The input and output of the target convolutional neural network are certain, and the model obtained for the target convolutional neural network can be easily replaced. This provides enough room for improving the performance of expression recognition and analysis for the entire system if the hardware conditions are met.

[0143] The emotion classification given by the target convolutional neural network is essentially a probability distribution, which is regarded as the emotional distribution of the subject at the moment of the expression frame picture. The probability distribution of each expression frame picture is displayed on the chart according to the time series, and the specific picture played at a specific time by the video played by the terminal device is used to give the subject visual stimulation. The obtained emotion distribution change curve is the emotion evaluation result of the subject.

[0144] Through this result, both psychologists and managers related to the subjects (such as school teachers, judicial personnel, etc.) can obtain the subjects' emotional reports and provide targeted diagnosis, education, assistance, etc. to the subjects based on the contents of the report.

[0145] In a second aspect, the present application provides an emotion assessment system, which is applied to the above-mentioned emotion assessment method, and the system comprises:

[0146] An acquisition unit, used to acquire video frames of a video played within a preset time period, and expression frames of a user's micro-expression changes when watching a video, wherein the expression frame represents a user's facial expression recorded by a frame of image;

[0147] A matching unit, used for matching the video frame with the expression frame to obtain data to be analyzed;

[0148] An analysis unit, used to input the data to be analyzed into a target convolutional neural network to obtain micro-expression data analysis results;

[0149] The result unit is used to obtain the user's emotion evaluation result based on the micro-expression data analysis result.

[0150] In a third aspect, the present application provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the aforementioned emotion assessment method are implemented.

[0151] In a fourth aspect, the present application provides a computer program, which, when executed by a processor, implements the steps of the aforementioned emotion assessment method.

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0153] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0154] The protection scope of the present disclosure is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A method for evaluating emotions, characterized in that: Methods include: Obtain video frames of the video played within a preset time period, and expression frames of the user's micro-expression changes when watching the video, wherein the expression frame represents the user's facial expression recorded by a frame of image; Matching the video frame with the expression frame to obtain data to be analyzed; Inputting the data to be analyzed into the target convolutional neural network to obtain micro-expression data analysis results; Based on the micro-expression data analysis results, the user's emotion evaluation results are obtained.

2. The emotion evaluation method according to claim 1, characterized in that: The step of obtaining video frames of the video played within a preset time period, and expression frames of the user's micro-expression changes when watching the video, includes: Acquire a pre-built video to obtain a video frame, wherein the video at least includes a strong emotional stimulation element that can cause a change in the user's mentality; According to the time when the video is played, the micro-expression changes of the user's face when watching the video are obtained to obtain expression frames.

3. The emotion evaluation method according to claim 1, characterized in that: The step of matching the video frame with the expression frame to obtain the data to be analyzed includes: Assigning a timestamp to each of the video frames and each of the expression frames to obtain the video frames and the expression frames having the timestamp; According to the video frame and the expression frame having the timestamp, corresponding matching is performed between the video frame and the expression frame to obtain data to be analyzed.

4. The emotion evaluation method according to claim 1, characterized in that: The step of inputting the data to be analyzed into a target convolutional neural network to obtain micro-expression data analysis results comprises: Performing alignment and clipping preprocessing on the data to be analyzed to obtain target analysis data; Analyzing the target data, performing feature extraction, and obtaining micro-expression feature information; The micro-expression feature information is input into the target convolutional neural network to obtain micro-expression data analysis results.

5. The emotion evaluation method according to claim 4, characterized in that: The step of analyzing the target data, extracting features, and obtaining micro-expression feature information comprises: Performing optical flow motion intensity analysis on the expression frames in the target analysis data to determine the peak frame with the highest intensity of each micro-expression; Taking the peak frame as the center, optical flow field calculation is performed with a preset number of preceding and following frames to obtain optical flow field information in the preceding and following frames of the peak frame, wherein the preceding and following frames represent a preset number of frames before the peak frame and a preset number of frames after the peak frame; The micro-expression feature information is obtained according to the optical flow field information.

6. The emotion evaluation method according to claim 5, characterized in that: The step of inputting the micro-expression feature information into a target convolutional neural network to obtain a micro-expression data analysis result comprises: Construct a convolutional neural network model with a 3D CNN architecture and obtain an initial convolutional neural network; Using historical micro-expression data to train an initial convolutional neural network to obtain the target convolutional neural network; The micro-expression feature information is input into the target convolutional neural network to obtain micro-expression data analysis results.

7. The emotion evaluation method according to claim 1, characterized in that: The step of obtaining the user's emotion evaluation result according to the micro-expression data analysis result comprises: According to the micro-expression data analysis results, the probability of predicting micro-expressions is obtained; According to the probability of predicting micro-expressions, the user's emotion evaluation result is obtained.

8. An emotion assessment system, characterized in that: The emotion assessment method applied to any one of claims 1 to 7, wherein the system comprises: An acquisition unit, used to acquire video frames of a video played within a preset time period, and expression frames of a user's micro-expression changes when watching a video, wherein the expression frame represents a user's facial expression recorded by a frame of image; A matching unit, used for matching the video frame with the expression frame to obtain data to be analyzed; An analysis unit, used for inputting the data to be analyzed into a target convolutional neural network to obtain micro-expression data analysis results; The result unit is used to obtain the user's emotion evaluation result based on the micro-expression data analysis result.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the emotion assessment method described in any one of claims 1 to 7 are implemented.

10. A computer program, characterized in that When the computer program is executed by a processor, the steps of the emotion assessment method described in any one of claims 1 to 7 are implemented.

Images