Method and device for evaluating real emotion and video consultation system

Abstract

The invention provides a method and device for evaluating real emotion and a video consultation system, and the method comprises the steps: obtaining audio and video data of a testee, and separating the audio and video data into an audio stream and a video stream; respectively carrying out framing processing on the audio stream and the video stream to obtain a plurality of voice frames and a plurality of video frames; performing face detection preprocessing on the video frame to obtain a face image, inputting the face image into an expression recognition model to obtain expression confidence, processing through an expression analysis method to obtain expression comprehensive confidence, inputting an audio frame into an emotion recognition model to obtain emotion confidence, and processing through an emotion analysis method to obtain expression comprehensive confidence; the emotion comprehensive confidence is obtained; according to the emotion analysis method, based on emotion abrupt change, an audio and video frame rate FPS and the emotion confidence coefficient, similarity calculation is carried out on the expression comprehensive confidence coefficient and a first standard confidence coefficient, and similarity calculation is carried out on the emotion comprehensive confidence coefficient and a second standard confidence coefficient. Through the expression recognition model and the emotion analysis method, the accuracy of evaluating the real emotion of the testee is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for evaluating real emotions and a video consultation system. Background Art

[0002] At present, the psychological analysis evaluation method is mainly based on the psychological test scale. Professionals conduct face-to-face question-and-answer sessions with the test subjects. Professionals collect the test subjects' answers to each question in the psychological test scale, as well as the test subjects' performance status and subconscious reactions during the question-and-answer process, and comprehensively analyze this information based on experience to obtain the test subjects' psychological analysis results. This psychological testing method relies on the experience of professionals, and the accuracy of psychological testing is unstable. When there are a large number of test subjects, the test efficiency is low, and the accuracy of psychological testing is also unstable. If the test subject is familiar with the test scale, there is a possibility that the true intention is hidden subjectively, which ultimately affects the results of psychological test analysis.

[0003] At present, the common professional mental health counseling services are still carried out one-on-one by professional psychologists based on psychological test scales, and some units have set up psychological huts and other equipment. The number of professional mental health service personnel is far from meeting the growing demand, especially in areas where educational resources are relatively scarce. In addition, there is a shortage of facilities and equipment for mental health education and counseling services, which limits the popularity and effectiveness of mental health services.

[0004] To address these problems, the academic community uses technology to develop and promote mental health-related applications and online services, and uses technical means such as artificial intelligence and big data analysis to improve the accessibility and efficiency of mental health services.

[0005] In order to improve the analysis results of the psychological test scale, the facial expression recognition and emotional state of the user are focused on by using facial analysis technology based on artificial intelligence (AI). With the rapid development of AI, the expression recognition model FER trained based on the deep learning Bible network is getting better and better. However, in fact, there is the possibility of suppression and performance of expressions. For example, professional actors can perform various expressions and express different emotional states. The recognition results of the expression recognition model FER used for the analysis of the psychological test scale have limited effect on the improvement of the results. Therefore, psychological experts put forward the concept of micro-expressions and believe that micro-expressions can truly reflect the true emotional state of the test subject. However, the micro-expression recognition model based on AI has poor accuracy due to the lack of data and difficulty in collection.

[0006] At present, some smart cabins are built using modern artificial intelligence technology, VR technology and medical monitoring equipment, and multi-dimensional and multi-modal feature data of the subjects are collected at the same time, such as collecting audio and video data of the subjects through cameras and microphones, measuring facial temperature in infrared through VR helmets, measuring pulse and blood pressure with bracelets, and measuring other physiological characteristics with special chairs. Then, these data are comprehensively analyzed and combined with psychological test scales to finally give an assessment of the mental health status of the subjects. However, this kind of facility is expensive and not all units can configure it. Even if it is configured, the number is limited and cannot meet the needs of full coverage screening.

[0007] Therefore, there is an urgent need for a data collection and analysis method that is low-cost, fully covered, and efficient and accurate in identifying the true emotions of the subjects. Summary of the invention

[0008] The purpose of the present invention is to provide a method, device and video consultation system for evaluating real emotions. The present invention aims to solve the problems of poor stability of manual consultation and evaluation of mental health, difficulty in identifying real emotions, difficulty in building micro-expression recognition models, and high cost of multimodal physiological sign detection. A method for evaluating real emotions is proposed. The expression confidence and emotion confidence obtained by the expression recognition model and emotion recognition model can evaluate the real emotions of the person being tested. The device based on the method and the video consultation system based on AI and digital human disclosed by the present invention have low construction cost, high analysis accuracy, good stability, and can be popularized.

[0009] To achieve the above object, the present invention provides a method for evaluating real emotions, comprising the following steps:

[0010] Acquire the audio and video data of the test subject, separate the audio and video data into an audio stream and a video stream; perform frame processing on the audio stream and the video stream to obtain a plurality of voice frames and a plurality of video frames;

[0011] The video frame is preprocessed with face detection to obtain a face image, and the face image is input into an expression recognition model to obtain expression confidence; the expression confidence is a p-dimensional vector (a1, a2, ...a p ), each vector component a i is the expression component of the expression recognition model, each expression component is a probability value predicted to be the expression, and the expression confidence is processed by the expression analysis method to obtain the expression comprehensive confidence; the expression analysis method obtains the expression comprehensive confidence based on expression mutation, audio and video frame rate FPS and the expression confidence;

[0012] The speech frame is input into the emotion recognition model to obtain the emotion confidence; the emotion confidence is a p-dimensional vector (b1, b2, ...b p ), each vector component b iis the emotion component of the emotion recognition model, each emotion component is a probability value, and the emotion confidence is processed by the emotion analysis method to obtain the emotion comprehensive confidence; the emotion analysis method obtains the emotion comprehensive confidence based on emotion mutation, audio and video frame rate FPS and the emotion confidence;

[0013] The similarity between the comprehensive confidence of expression and the first standard confidence is calculated, and the similarity between the comprehensive confidence of emotion and the second standard confidence is calculated to evaluate the real emotion of the test subject.

[0014] Optionally, constructing the expression recognition model includes: obtaining a data set, the data set including a certain number of photos, the photos including faces, labeling the photos according to p types of expression categories, the number of labeled photos meeting the recognition accuracy requirement and the labeled data of each category being balanced;

[0015] Process the photo to determine the face area, input the data of the face area into the face recognition model, and extract the expression features through convolution and pooling operations; the expression features are reduced to p categories through information processing; the output data of the expression recognition model is a p-dimensional vector, and the expression category corresponding to the component with the largest probability value in the p-dimensional vector is used as the expression result predicted by the expression recognition model.

[0016] Optional methods for assessing true sentiment include:

[0017] Get video frame rate FPS and resolution information, and calculate frame interval time

[0018] The expression recognition model FER is a P classification model. In each complete semantic expression period, m (m is greater than or equal to 1) expressions are presented. The FER model output at the i-th expression mutation is CS i , i starts from 0, when the expression undergoes a sudden change When there are frames, the comprehensive confidence CCS of the i-th expression i The maximum value is recorded as:

[0019] Then CCS i Reduce frame by frame, expressed as follows:

[0020] Among them CS i is the confidence of the first expression detected when a sudden change occurs, n is the number of frames from the last sudden change to the next sudden change, CS f is the confidence of the result of FER recognition of the fth frame, f is the frame number during the complete semantic expression, and f iIt represents the first frame of the ith expression mutation. The comprehensive expression confidence during the entire semantic expression period is:

[0021] CCS is normalized, and both CS and CCS are P-dimensional vectors.

[0022] Optionally, a data set is obtained, the data set includes recordings of multiple emotions, and the P expressions corresponding to the emotions for constructing the expression recognition model are annotated, the number of annotated recordings meets the requirement of emotion recognition accuracy, and the data of each category is balanced after annotation;

[0023] Remove background noise and echo from the recorded data, unify the format of the recorded data, and make the sampling rate, data bit number and channel parameters of the recorded data consistent;

[0024] The Mel-frequency cepstral coefficient (MFCC) acoustic features are obtained, and the frequency domain features are extracted using a convolutional neural network (CNN). The time series information is processed by a long short-term memory (LSTM), and finally processed and reduced to a p-dimensional vector through a fully connected layer. The emotion recognition model outputs a p-dimensional vector, and the emotion classification corresponding to the component with the largest probability value is used as the emotion result predicted by the emotion recognition model.

[0025] Optionally, the sentiment analysis method includes:

[0026] Get the video frame rate FPS information and calculate the frame interval time

[0027] The emotion recognition model is a P classification model. In each complete semantic expression period, m (1≤m≤p) emotions are presented. When the i-th emotion mutation occurs, the output of the emotion recognition model is CS i , i starts from 0, when the emotion changes suddenly The comprehensive confidence CCS of the i-th emotion is i The maximum value is recorded as:

[0028] Then CCS i Reduce frame by frame, expressed as follows: Among them CS i is the confidence of the first detected emotion when an emotion mutation occurs, n is the number of frames from the last emotion mutation to the next emotion mutation, CS f is the confidence of the recognition result of the emotion recognition model for the fth frame, f is the frame number during the complete semantic expression, and f i is the first frame of the i-th emotion mutation, and the comprehensive emotion confidence during the entire semantic expression is:

[0029] CCS is normalized, CS and CCS are both P-dimensional vectors, and the normalized comprehensive emotion confidence is similarly calculated with the second standard confidence.

[0030] Optionally, the method for calculating the standard confidence level includes:

[0031] The standard confidence is the median of the comprehensive confidence of the statistical sample, representing the reference value of normal emotional expression. The reference value of the comprehensive expression confidence is the first standard confidence, and the reference value of the comprehensive emotion confidence is the second standard confidence.

[0032] The standard confidence level can be taken as the combined confidence mean or the mean of the middle 50% of the distribution.

[0033] Optionally, standard confidence levels are used to evaluate mental health issues, and different mental health issues define different standard confidence levels based on statistical results.

[0034] The present invention provides a device for evaluating real emotions, characterized by comprising: an acquisition module for acquiring audio and video data of a test subject, and separating the audio and video data into an audio stream and a video stream;

[0035] A processing module, performing frame processing on the audio stream and the video stream to obtain multiple voice frames and multiple video frames;

[0036] Performing face detection preprocessing on the video frame to obtain a face image, and inputting the face image into an expression recognition model to obtain expression confidence;

[0037] The expression confidence is a p-dimensional vector (a1, a2, ...a p ), each vector component a i is the expression component of the expression recognition model, each expression component is a probability value predicted to be the expression, and the expression confidence is processed by the expression analysis method to obtain the expression comprehensive confidence;

[0038] The expression analysis method obtains the comprehensive confidence of the expression based on the expression mutation, the audio and video frame rate FPS and the expression confidence;

[0039] Inputting the speech frame into an emotion recognition model to obtain emotion confidence;

[0040] The emotion confidence is a p-dimensional vector (b1, b2, ...b p ), each vector component b i is the emotion component of the emotion recognition model, each emotion component is a probability value, and the emotion confidence is processed by the emotion analysis method to obtain the emotion comprehensive confidence;

[0041] The emotion analysis method obtains the comprehensive emotion confidence based on emotion mutation, audio and video frame rate FPS and the emotion confidence;

[0042] The evaluation module performs similarity calculation between the comprehensive confidence of the expression and the first standard confidence, and performs similarity calculation between the comprehensive confidence of the emotion and the second standard confidence, so as to evaluate the real emotion of the test subject.

[0043] The present invention provides a video consultation system based on AI and digital human, characterized in that the system includes, in addition to the device of claim 8, further comprising:

[0044] A human-computer interaction module, including a web interface for interacting with an administrator user and a web video chat interface for interacting with a terminal user, wherein the web video chat interface collects user audio and video data through a camera and a microphone and uploads the data to a server, and the device receives the audio and video data;

[0045] An AI model module, which provides the device with the required AI functions and dynamically generates audio and video data for the human-computer interaction module;

[0046] Data management module, used to manage users and data processing result data;

[0047] The digital human management module is used to create digital humans and maintain character attributes, images, and voice features.

[0048] Optionally, the AI ​​model module includes an expression recognition model, an emotion recognition model, a speech recognition model, an LLM model, a text-to-speech model and a digital human model, the expression recognition model, the emotion recognition model and the speech recognition model;

[0049] The expression recognition model, emotion recognition model, and speech recognition model provide expression recognition, emotion recognition, and speech semantic recognition functional services to the device respectively;

[0050] The LLM model automatically generates text content according to the interaction needs between the digital human and the user;

[0051] The text-to-speech model generates a speech stream from the text content;

[0052] The voice stream drives the digital human model to generate a video stream, and generates an audio and video stream together with the voice stream;

[0053] The generated audio and video streams are presented by the human-computer interaction module.

[0054] Optionally, the digital human management module includes:

[0055] The digital human creation module is used by users to create digital humans, including uploading the digital human image, naming the digital human, defining the digital human role attributes, and defining the voice features;

[0056] The AI ​​digital human model generates a digital human image based on a speech video or a photo of a person uploaded by a user;

[0057] The naming requires that the named name uniquely identifies the digital person;

[0058] The digital human role attributes refer to the digital human identity information, including one or more of occupation, gender, age, and language;

[0059] The language includes Mandarin Chinese, Chinese dialects, or foreign languages;

[0060] The voice features match the digital human character attributes. Technical Effects

[0061] In summary, the present application can obtain expression confidence based on an expression recognition model, obtain emotion confidence based on an emotion recognition model, adopt the expression analysis method to obtain the comprehensive expression confidence based on expression mutation, audio and video frame rate FPS and the expression confidence, adopt the emotion analysis method to obtain the comprehensive emotion confidence based on emotion mutation, audio and video frame rate FPS and the emotion confidence, compare the comprehensive expression confidence and the comprehensive emotion confidence with the corresponding standard confidence for similarity, thereby realizing the evaluation of real emotions, with the characteristics of accurate evaluation results, low cost and high evaluation efficiency, and the evaluation effect is significant for application scenarios such as psychological scale testing, psychological intervention, lie detection, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0063] Figure 1 A flowchart of a method for assessing true emotion.

[0064] Figure 2 Data flow diagram for a method to assess true sentiment.

[0065] Figure 3 The flowchart of the comprehensive confidence algorithm of expression is a method for evaluating real emotions.

[0066] Figure 4 This is a layered architecture diagram of a video consultation system based on AI and digital humans. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0069] The specific implementation of the present invention is described in detail below in conjunction with a specific system embodiment. Embodiment 1

[0070] This embodiment provides a method for evaluating real emotions. The expression recognition model refers to a computer algorithm or system that can automatically separate a specific expression state from a given static image or dynamic video sequence and judge the psychological emotions of the recognized object based on it. This type of model is mainly based on technologies in the fields of computer vision, machine learning, and artificial intelligence. It realizes the recognition and classification of facial expressions by extracting and analyzing expression features in face images or video frames.

[0071] The expression analysis method comprises the following steps: The audio and video data of the test subject is obtained, and the audio and video data is separated into an audio stream and a video stream. Specifically, the camera and microphone data are collected through the browser client human-computer interaction interface to synthesize the audio and video stream and upload it to the server. The server data processing device receives the audio and video stream and separates the audio and video stream into an audio stream and a video stream; the audio stream and the video stream are respectively framed to obtain multiple voice frames and multiple video frames;

[0072] The video frame is subjected to face detection preprocessing to obtain a face image, the face image is converted into a three-dimensional matrix (x, y, c), where (x, y) is a pixel coordinate and c is a color value, and the three-dimensional matrix data is input into an expression recognition model to obtain expression confidence;

[0073] The speech frame is resampled to obtain monophonic PCM data with a sampling rate of 16 KHz, a sampling size of 16 bits, and is input into an emotion recognition model to obtain emotion confidence.

[0074] When applied to the psychological scale test scenario, the digital human and the test subject have an audio and video chat. The test subject's audio and video data are obtained, and the expression confidence is obtained by the expression recognition model. The expression confidence is a p-dimensional vector (a1, a2, ...a p ), each vector component a i The prediction of the expression recognition model is the probability value of its corresponding expression, and satisfies Assuming that the facial expression recognition model is an 8-classification model, and the facial expressions are divided into 8 categories: anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral, then the facial expression confidence is an 8-dimensional vector, and its components a1, a2, a3, a4, a5, a6, a7, and a8 correspond to the probability values ​​of the predicted expressions of anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral, respectively. After the facial expression confidence is processed by the facial expression analysis method, the comprehensive facial expression confidence of the 8-dimensional vector is obtained. The comprehensive facial expression confidence is a p-dimensional vector (a'1, a'2, ... a' p ), each dimension corresponds to an expression, a total of p expressions, each component a i Represents the credibility of the corresponding expression. The comprehensive confidence of expression is used to evaluate the credibility of various expressions that appear during the complete semantic expression, and the credibility can be used to evaluate the real emotion. If the expression recognition model is an 8-classification model, after being processed by the expression analysis method, the comprehensive confidence of expression is (1, 0.0013, 0.0103, 0.0009, 0.0020, 0.0021, 0.0102, 0.0022, 0.2987). The analysis result shows that during the user's speech (i.e., during the semantic expression), the user first showed a momentary angry micro-expression (a'1). The user wanted to conceal the anger, so he subjectively suppressed the angry micro-expression. At this time, the user's expression showed a neutral expression (a'8 = 0.2987). During this semantic expression, anger is his true emotion, and the vector component a'1 (probability value is 1) represents the micro-expression he wants to conceal. After being processed by the expression analysis method, the user's real expression and the various expressions that appear during semantic expression can be easily and accurately judged (assuming that the expression corresponding to a probability value greater than 0.1 is the expression presented, and the expression with a probability value less than 0.1 can be ignored).

[0075] Assume that after being processed by the expression analysis method, the comprehensive confidence of the expression is (1, 0.9513, 0.0103, 0.0009, 0.0020, 0.0021, 0.0102, 0.2022, 0.2987). The analysis result indicates that during the user's speech (i.e., during the semantic expression period), the user's micro-expressions (expressions with probability values ​​close to 1) include anger (subjectively changed to contempt corresponding to 0.2022) and disgust (subjectively changed to neutral expression). The number of expressions close to 1 (1 and 0.9513, the number is 2) and the probability value is between 0.2- When the number between 0.5 (0.2022 and 0.2987, the number is also 2) is the same, the expressions close to 1 (1 (anger) and 0.9513 (disgust)) are considered to be micro-expressions to be concealed. If the number of expressions close to 1 is inconsistent with the number between 0.2 and 0.5, the expression close to 1 is not a micro-expression to be concealed, but the user's real expression. For example: after being processed by the expression analysis method, the comprehensive confidence of the expression is (0.8900, 0.0013, 0.0103, 0.0009, 0.0020, 0.0021, 0.0102, 0.9102, 0.9087). The analysis result shows that during the user's speech (i.e., during the semantic expression period), the user's expressions of anger, contempt, and neutrality are all real expressions. The user did not subjectively conceal these expressions, nor did he want to conceal a certain micro-expression.

[0076] By introducing the real emotion evaluation method to obtain the comprehensive confidence to judge the user's real expression or emotion, it is very effective for application scenarios such as psychological scale testing, psychological intervention, and lie detection. It is an efficient and accurate solution for analyzing the real emotions in user conversations.

[0077] The emotion analysis method is based on the same principle as the expression analysis method. The emotion confidence is a p-dimensional vector (b1, b2, ...b p ), each vector component b i The prediction of the emotion recognition model is the probability value of the corresponding emotion, and satisfies Assuming that the emotion recognition model is an 8-classification model, the emotions are divided into 8 categories: anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral. Then the emotion confidence is an 8-dimensional vector, and its components b1, b2, b3, b4, b5, b6, b7, and b8 are the probability values ​​of anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral emotions. The expressed emotion confidence is processed by the emotion analysis method to obtain the emotion comprehensive confidence; the emotion comprehensive confidence is a p-dimensional vector (b'1, b'2, ...b' p ), each dimension corresponds to an emotion, there are p emotions in total, and each component b' i Represents the credibility of the corresponding emotion. The comprehensive confidence is used to evaluate the credibility of various emotions that appear during the complete semantic expression, and the real emotion can be evaluated based on this credibility.

[0078] If the emotion recognition model is an 8-classification model, the comprehensive confidence of emotion is (1, 0.0023, 0.0143, 0.0019, 0.0021, 0.0023, 0.0132, 0.0032, 0.4924), it means that during the semantic expression, micro-emotion anger appeared, and then it was expressed as neutral emotion after subjective suppression. During this semantic expression, anger was its true emotion, and the vector component represents the influence of its corresponding emotion. Through the emotion analysis method, it is easy and accurate to judge the real emotion and the multiple emotions that appear during the semantic expression. The emotion analysis method can be used alone or in conjunction with the expression analysis method. The latter has a better effect of mutual verification. Similar to the expression analysis method, it is also effective for application scenarios such as psychological scale testing, psychological intervention, and lie detection, and only requires a microphone, which is a lower-cost solution for analyzing real emotions.

[0079] Optionally, an existing expression recognition model that meets the classification requirements may be used. If the expression recognition model is constructed, the following steps are included:

[0080] Get a data set, the data set includes a certain number of photos, the photos include faces, the photos are annotated according to p types of expression categories, the number of annotated photos meets the recognition accuracy requirements and the data of each category is balanced after annotation. There are many public expression data sets, which are usually used for computer vision and machine learning tasks such as facial expression recognition and sentiment analysis. The following are some public expression data sets: RAF (Real-world Affective Faces) data set, containing 29,672 images, 7 basic expressions and 12 compound expressions, each image also provides 5 accurate facial key points. EmotioNet data set, containing 950,000 images, including basic expressions, compound expressions, and expression unit annotations. AffectNet data set, more than 420,000 images, annotation types include expression type and amplitude, including 8 basic expressions.

[0081] Clean and annotate the data set, and annotate it according to eight types of expressions: anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral. If the amount of data is insufficient or the classification is unbalanced, these will affect the model recognition effect. You can collect it yourself, or ask someone to perform the required emotions and shoot, and then manually calibrate and annotate;

[0082] The neural network structure of the expression recognition model is designed. The model input is a 3D matrix (x, y, c), where (x, y) is the pixel coordinate and c is the color value. The face recognition model determines the face area of ​​the input data, and the data of the face area is subjected to convolution and pooling operations to extract the expression features. The expression features are processed by information, that is, multiple fully connected layers are processed, and the dimension is reduced to p categories. The expression recognition model outputs a p-dimensional vector, and the expression category corresponding to the component with the largest probability value is used as the expression result predicted by the expression recognition model.

[0083] If facial expressions are divided into eight categories, namely anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality, the model output is an 8-dimensional vector, corresponding to the predicted probability values ​​of the eight facial expressions, and the expression corresponding to the component with the highest probability value is the model predicted expression. For example, if the model output is (0.1023, 0.0102, 0.0183, 0.8023, 0.0523, 0.0123, 0.0023, 0.0003), where 0.1023+0.0102+0.0183+0.8023+0.0523+0.0123+0.0023+0.0003=1, and the probability value 0.8023 is the largest. This probability value is the 4th component of the vector, and the 4th expression is happy. The model predicts that the expression is happy, and its confidence is 8.8023.

[0084] For the expression analysis method, the data analysis device is constructed, and the steps are as follows: Get video frame rate FPS and resolution information, and calculate frame interval time The expression recognition model FER is a P classification model. In each complete semantic expression period, m (m is greater than or equal to 1) expressions are presented. The FER model output at the i-th expression mutation is CS i , i starts from 0.

[0085] According to research, the main difference between micro-expressions and facial expressions is that they last for a short time. Micro-expressions last for about 0.2 seconds and no more than 0.5 seconds. If they last for more than 0.5 seconds, they become facial expressions. When the expression suddenly changes after 0.2 seconds, the credibility of the expression reaches the maximum value, and the number of frames at this time is The comprehensive confidence CCS of the i-th expression i The maximum value is recorded as:

[0086] Then CCS i After 0.5 seconds, or 0.5 FPS, the expression credibility is close to 0. It can be expressed as follows: Among them CS i is the confidence of the first expression detected when a sudden change occurs, n is the number of frames from the last sudden change to the next sudden change, CS f is the confidence of the result of FER recognition of the fth frame, f is the frame number during the complete semantic expression, and f i represents the first frame of the ith expression mutation, where CS i is the credibility of the model output, Is based on the duration of CS i correction.

[0087] If it is a normal expression, the correction part is close to 0, and the final comprehensive confidence CCS i and CS i close, that is, this algorithm has no effect on normal expressions; if it is a micro-expression, the correction part is magnified by more than 3 times because the micro-expression lasts for a short time, plus the original CS i , the final comprehensive confidence CCS i It's CS i The confidence level of the expression corresponding to i is more than 4 times that of the corresponding expression i, which highlights the influence of micro-expressions.

[0088] The comprehensive expression confidence during the entire semantic expression is: CCS is normalized, and the processed comprehensive confidence CCS represents the credibility of multiple expressions during semantic expression.

[0089] The comprehensive confidence is used for psychological test scale evaluation. The system asks the user psychological test scale questions through the digital human. The credibility of various expressions that appear during the complete semantic expression of the user's voice answer is the comprehensive confidence of the expression, which can be used to evaluate the real expression. If the expression recognition model is an 8-classification model, the comprehensive confidence of the expression is (1, 0.0023, 0.0143, 0.0019, 0.0021, 0.0023, 0.0132, 0.0032, 0.4924), which means that during the semantic expression, a micro-expression - anger appeared, and then it was subjectively suppressed and showed a neutral expression. In this semantic expression period, anger is its real expression, and the vector component represents the influence of its corresponding expression. Through this method, it is easy and accurate to judge the real expression of the test subject when answering the test scale questions and the various expressions that appear during the semantic expression.

[0090] Optionally, constructing the emotion recognition model includes: Obtain a dataset containing recordings of multiple emotions. Public sound emotion recognition datasets that can be downloaded include RAVDESS, SAVEE, EMO-DB, IEMOCAP, etc. These datasets usually contain recordings of multiple emotions.

[0091] According to the P kinds of expressions used to construct the expression recognition model, the corresponding emotions are labeled, and the number of labeled recordings meets the requirements for emotion recognition accuracy and the data of each category is balanced after labeling. If the eight emotions of anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality are labeled, it is an 8-category model. If the amount of data is insufficient or the classification is unbalanced, these will affect the recognition effect of the model. You can collect it yourself, or ask someone to perform the required emotions for recording, and then manually calibrate and label it. Note: Micro-expressions and micro-emotions cannot be performed, and data sets are difficult to collect and record. This determines that the construction of micro-expression and micro-emotion recognition models is difficult and the recognition effect is poor. For this reason, the present application discloses a method, device, and system for analyzing real emotions based on an expression recognition model, which achieves the best results at the lowest cost and has practical promotion significance.

[0092] Remove the background noise and echo of the recorded data, unify the format of the recorded data, and make the sampling rate, data bit number and channel parameters of the recorded data consistent. Here, the recorded data needs to be resampled by a resampling algorithm to obtain a sampling rate of 16KHz, a sampling size of 16bit, and monophonic PCM data;

[0093] Acquire Mel-frequency cepstral coefficient (MFCC) acoustic features, and use CNN to extract frequency domain features; process time series information through LSTM, and finally process and reduce the dimension to a p-dimensional vector through a fully connected layer; the emotion recognition model outputs a p-dimensional vector, and the emotion classification corresponding to the component with the largest probability value is used as the emotion result predicted by the emotion recognition model.

[0094] For example, multiple fully connected layers are used to reduce the dimension to a one-dimensional matrix, i.e., a vector, and then the vector is reduced to an 8-dimensional vector through multiple fully connected layers. The sum of the components of this 8-dimensional vector is 1, and the emotion corresponding to the dimension where the component with the largest probability value is located is the emotion predicted by the model.

[0095] Optionally, for sentiment analysis methods, include: Get the video frame rate FPS information and calculate the frame interval time

[0096] The emotion recognition model is a P classification model. In each complete semantic expression period, m (1≤m≤p) emotions are presented. When the i-th emotion mutation occurs, the output of the emotion recognition model is CS i , i starts from 0, when the emotion changes suddenly The comprehensive confidence CCS of the i-th emotion is i The maximum value is recorded as:

[0097] Then CCS i Reduce frame by frame, expressed as follows: Among them CS i is the confidence of the first detected emotion when an emotion mutation occurs, n is the number of frames from the last emotion mutation to the next emotion mutation, CS f is the confidence of the recognition result of the emotion recognition model for the fth frame, f is the frame number during the complete semantic expression, and f i is the first frame of the ith emotion mutation, and the comprehensive expression confidence during the entire semantic expression period is:

[0098] CCS is normalized; CS and CCS are both P-dimensional vectors.

[0099] CS is the output of the emotion recognition model, and the sum of its components is 1. The emotion corresponding to the dimension with the largest component value is the emotion predicted by the model; while the sum of CCS components is not 1. Each component of CCS represents the influence of its corresponding emotion. CCS records the influence of P emotions that appear during the semantic expression. Some emotions have strong influence, such as micro-emotions, and their influence is the strongest, which is 1 or close to 1; some emotions do not appear during the semantic expression, and the corresponding component value is 0 or close to 0; the influence of normal emotional expression remains unchanged, but micro-emotions appear during the semantic expression, and the emotions after micro-emotions are weakened to between 0 and 0.5. Therefore, CCS can simplify the evaluation of real emotions, with high efficiency, good effect, low cost and easy use.

[0100] Optional calculation method for standard confidence level, including:

[0101] Standard confidence: It is the median of the comprehensive confidence of the statistical sample, representing the reference value of normal emotional expression; the standard confidence can be the mean of the comprehensive confidence or the mean calculated from the middle 50% of the distribution. The standard confidence is to use the comprehensive confidence in specific practical application fields, which is convenient for evaluating the reference set by humans.

[0102] Optionally, standard confidence is used to evaluate mental health problems. Different mental health problems define different standard confidences based on statistical results. For example, when a normal person answers questions on a psychological test scale, several combinations of expressions may appear. The median of the comprehensive confidence can be classified and counted according to the expression combination as several standard confidences for these combinations. In actual application, the similarity between the comprehensive confidence of the test subject and these standard confidences is calculated. The one with the highest similarity to the standard confidence can be evaluated as a normal healthy person of the corresponding category.

[0103] For example, regarding depression, when patients with different degrees of depression answer questions on the psychological test scale, they may have several combinations of expressions. The median of the comprehensive confidence can be classified and counted according to the expression combination as several standard confidences for these combinations. If the sample size is small, the mean calculated from the middle 50% of the comprehensive confidence distribution can be taken. In actual application, the similarity between the comprehensive confidence of the test subject and the standard confidence of different degrees of depression is calculated. The one with the highest similarity to the standard confidence can be evaluated as depression of the corresponding category. This method can be iterated repeatedly with increasing accuracy. Embodiment 2

[0104] This embodiment provides a device for evaluating real emotions. In a psychological scale test scenario, the device is installed on an independent server to provide data analysis functions for other modules of the system. It is characterized by:

[0105] An acquisition module is used to acquire the audio and video data of the test subject and separate the audio and video into an audio stream and a video stream; a processing module is used to perform frame processing on the audio stream and the video stream to obtain multiple voice frames and multiple video frames;

[0106] The video frame is preprocessed with face detection to obtain a face image, and the face image is input into an expression recognition model to obtain expression confidence; the expression confidence is a p-dimensional vector (a1, a2, ...a p ), each vector component a i is the expression component of the expression recognition model, each expression component is a probability value, and the expression confidence is processed by the expression analysis method to obtain the expression comprehensive confidence;

[0107] The expression analysis method obtains the comprehensive confidence of the expression based on the expression mutation, the audio and video frame rate FPS and the expression confidence;

[0108] The speech frame is input into the emotion recognition model to obtain the emotion confidence. The emotion confidence is a p-dimensional vector (a1, a2, ...a p ), each vector component a i is the emotion component of the emotion recognition model, each emotion component is a probability value, and the emotion confidence is processed by the emotion analysis method to obtain the emotion comprehensive confidence;

[0109] The emotion analysis method obtains the emotion comprehensive confidence based on emotion mutation, audio and video frame rate FPS and the emotion confidence. For details of the expression analysis method and the emotion analysis method, please refer to the description in the first embodiment.

[0110] The evaluation module calculates the similarity between the comprehensive confidence of the expression and the comprehensive confidence of the emotion and the standard confidence respectively, so as to evaluate the real emotion of the test subject. Embodiment 3

[0111] This embodiment is a psychological counseling system based on web video chat. The test subject can have an audio and video chat with a digital person of his / her choice, and the system can collect the test subject's audio and video data. The system server includes the device described in the first embodiment, which analyzes micro-expressions based on the expression recognition model, analyzes micro-emotions based on the emotion recognition model, and comprehensively compares the first standard confidence and the second standard confidence to evaluate the test subject's true emotions.

[0112] The psychological counseling system is used in school scenarios. Teachers can allocate accounts and digital human image collections in the background, or upload photos or videos of selfies taken by the head teacher or other personnel to create digital humans, and then complete the test within a limited time.

[0113] Students can conduct psychological tests on their PC, Pad or mobile phone at home, or in the computer room of the school, as long as they can access the network and access the system of this embodiment. Students can choose their favorite (trusted) digital human characters when taking the test, such as the head teacher, parents, friends, cartoon idols, etc.

[0114] The psychological counseling system includes: a human-computer interaction module, which includes a web interface for interacting with an administrator user and a web video chat interface for interacting with a terminal user. The web video chat interface collects audio and video data of the test subject through a camera and a microphone and uploads it to the server, and the device receives the audio and video data; an AI model module, which provides the required AI functions for the device and dynamically generates audio and video data for the human-computer interaction module; a data management module, which is used to manage users and data processing result data; a digital human management module, which is used to create digital humans and maintain character attributes, images, and voice features.

[0115] Optionally, the AI ​​model module includes: an expression recognition model, an emotion recognition model, a speech recognition model, an LLM model, a text-to-speech model and a digital human model; the expression recognition model, the emotion recognition model and the speech recognition model respectively provide the device with expression recognition, emotion recognition and speech semantic recognition functional services; the LLM model automatically generates text content according to the interaction needs between the digital human and the user; the text-to-speech model generates a speech stream from the above text content; the speech stream drives the digital human model to generate a video stream, and generates an audio and video stream together with the speech stream; the generated audio and video stream is presented by the human-computer interaction module;

[0116] Optionally, the digital human management module includes: a digital human creation module, in which a user creates a digital human, including uploading a digital human image, naming the digital human, defining digital human role attributes, and defining voice features; the AI ​​digital human model generates a digital human image based on a speech video or photo of the person uploaded by the user; the naming of the digital human requires a unique name; the digital human role attributes refer to the digital human identity information, including occupation, gender, age, and language; the language includes Mandarin Chinese, Chinese dialects, or foreign languages; the voice features match the digital human role attributes.

[0117] like Figure 4 The above is a logical architecture diagram of a video consultation system based on AI and digital human provided by an embodiment of the present invention. The data processing layer in the figure is a specific implementation of a method for evaluating real emotions of the present invention. The system implementation includes the following steps:

[0118] Prepare various AI models and deploy them as network services for system calls. Including LLM (large language model), ASR (speech recognition model), TTS (text-to-speech model), and digital human model. Deploy these models to servers with GPU acceleration, develop OPENAI-compatible API network interfaces, and provide network call services for the corresponding modules of the system to call.

[0119] Develop an image-based expression recognition model FER. Including:

[0120] The FER neural network structure is designed to extract features through multiple convolution and pooling operations. Then the information is processed through multiple fully connected layers and finally reduced to eight categories, corresponding to eight expressions: anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality. The model output is the maximum confidence value and the corresponding expression classification.

[0121] Get the expression dataset. There are many public expression datasets, which are usually used for computer vision and machine learning tasks such as facial expression recognition and sentiment analysis. Here are some public expression datasets: RAF (Real-world Affective Faces) dataset, containing 29,672 images, 7 basic expressions and 12 compound expressions, each image also provides 5 accurate facial key points. EmotioNet dataset, containing 950,000 images, including basic expressions, compound expressions, and expression unit annotations. AffectNet dataset, more than 420,000 images, annotation types include expression type and amplitude, including 8 basic expressions.

[0122] Clean and annotate the data set, and annotate it according to eight types of expressions: anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral. If the amount of data is insufficient or the classification is unbalanced, these will affect the model recognition effect. You can collect it yourself, or ask someone to perform the required emotions and shoot, and then manually calibrate and annotate.

[0123] Image data processing, through mature face detection technology Haar feature cascade classifier, FRT, etc., determine the face area, process the data into multi-dimensional vectors or matrices as model data preprocessing, and use the face image as model input.

[0124] FER training uses GPU-accelerated servers for training. During the training process, you may need to monitor the model convergence at any time, adjust the model and data, and more importantly, adjust the training parameters.

[0125] FER deployment,When the model converges and,indicators such as loss and test accuracy meet expectations, the,model can be deployed and an OPENAI-compatible API network service,interface can be developed for other modules to call.

[0126] Develop a speech-based emotion recognition model. Including:

[0127] The neural network structure of the emotion recognition model is designed, and the acoustic features such as Mel-frequency cepstral coefficients (MFCC) are extracted as input. The frequency domain features are extracted using CNN, and the time series information is processed through long short-term memory (LSTM). Then, the information is processed through multiple fully connected layers, and finally the dimension is reduced to eight categories, corresponding to eight emotions such as anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality. The model output is the maximum confidence value and the corresponding emotion classification.

[0128] Obtain speech datasets. Public sound emotion recognition datasets include RAVDESS, SAVEE, EMO-DB, IEMOCAP, etc. These datasets usually contain recordings of multiple emotions. Clean and annotate the data set, and annotate it according to eight emotions: anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality. If the amount of data is insufficient or the classification is unbalanced, these will affect the model recognition effect. You can collect it yourself, or ask someone to perform the required emotions and record them, and then manually calibrate and annotate them.

[0129] Emotion recognition model training uses GPU-accelerated servers for training. During the training process, you may need to monitor the model convergence at any time, and you may need to adjust the model and data, especially the training parameters.

[0130] Emotion recognition deployment,When the model converges and indicators such as loss and test accuracy meet expectations, the model can be deployed and an OPENAI-compatible API network service interface can be developed for other modules to call.

[0131] The method and device for evaluating real emotions and the video consultation system based on AI and digital humans disclosed in this application are applied to students' mental health exclusion. The test is easy to organize and implement, with high efficiency, good effect, accurate results, low cost, and can be fully promoted.

Claims

1. A method for evaluating true emotion, characterized in that include: Obtain the audio and video data of the test subject, and separate the audio and video data into audio stream and video stream. The audio stream and the video stream are subjected to frame processing to obtain a plurality of voice frames and a plurality of video frames, The video frame is preprocessed with face detection to obtain a face image, and the face image is input into an expression recognition model to obtain expression confidence. The expression confidence is a p-dimensional vector (a1, a2, ...a p ), each vector component a i is the expression component of the expression recognition model, each expression component is the probability value predicted as the expression, and the expression confidence is processed by the expression analysis method to obtain the expression comprehensive confidence. The expression analysis method obtains the comprehensive confidence of the expression based on the expression mutation, the audio and video frame rate FPS and the expression confidence; Input the speech frame into the emotion recognition model to obtain the emotion confidence, The emotion confidence is a p-dimensional vector (b1, b2, ...b p ), each vector component b i is the emotion component of the emotion recognition model, each emotion component is a probability value, and the emotion confidence is processed by the emotion analysis method to obtain the emotion comprehensive confidence. The emotion analysis method obtains the comprehensive emotion confidence based on emotion mutation, audio and video frame rate FPS and the emotion confidence. The similarity between the comprehensive confidence of expression and the first standard confidence is calculated, and the similarity between the comprehensive confidence of emotion and the second standard confidence is calculated to evaluate the real emotion of the test subject.

2. The method according to claim 1, characterized in that The method also includes constructing the expression recognition model, including: Obtain a data set, the data set includes a certain number of photos, the photos include faces, the photos are labeled according to p types of expression categories, the number of labeled photos meets the recognition accuracy requirement and the labeled data of each category is balanced; Processing the photo, determining the face area, inputting the data of the face area into the face recognition model, and extracting expression features through convolution and pooling operations; The expression features are reduced to p categories through information processing; The output data of the expression recognition model is a p-dimensional vector, and the expression classification corresponding to the component with the largest probability value in the p-dimensional vector is used as the expression result predicted by the expression recognition model.

3. The method according to claim 1, characterized in that The expression analysis method comprises: Get video frame rate FPS and resolution information, and calculate frame interval time The expression recognition model FER is a P classification model. In each complete semantic expression period, m (1≤m≤p) expressions are presented. The FER model output at the i-th expression mutation is CS i , i starts from 0, when the expression undergoes a sudden change When there are frames, the comprehensive confidence CCS of the i-th expression i Reaching the maximum CCS i,max , recorded as: Then CCS i Reduce frame by frame, expressed as follows: Among them CS i is the confidence of the first expression detected when a sudden change occurs, n is the number of frames from the last sudden change to the next sudden change, CS f is the confidence of the result of FER recognition of the fth frame, f is the frame number during the complete semantic expression, and f i It represents the first frame of the ith expression mutation. The comprehensive expression confidence during the entire semantic expression period is: CCS is normalized, CS and CCS are both P-dimensional vectors, and the similarity between the normalized comprehensive expression confidence and the first standard confidence is calculated.

4. The method according to claim 1, characterized in that Constructing the emotion recognition model includes: Obtain a data set, the data set containing recordings of multiple emotions, annotate the P types of expressions corresponding to the emotions for constructing the expression recognition model, the number of annotated recordings meets the requirement of emotion recognition accuracy and the data of each category are balanced after annotation; Remove background noise and echo from the recorded data, unify the format of the recorded data, and make the sampling rate, data bit number and channel parameters of the recorded data consistent; Obtain Mel-frequency cepstral coefficient (MFCC) acoustic features and use convolutional neural network (CNN) to extract frequency domain features; The time series information is processed through long short-term memory (LSTM), and finally processed through a fully connected layer and reduced to a p-dimensional vector; The output of the emotion recognition model is a p-dimensional vector, and the emotion classification corresponding to the component with the largest probability value is used as the emotion result predicted by the emotion recognition model.

5. The method according to claim 1, characterized in that The sentiment analysis method comprises: Get the video frame rate FPS information and calculate the frame interval time The emotion recognition model is a P classification model. In each complete semantic expression period, m (1≤m≤p) emotions are presented. When the i-th emotion mutation occurs, the output of the emotion recognition model is CS i , i starts from 0, when the emotion changes suddenly The comprehensive confidence CCS of the i-th emotion is i The maximum value is recorded as: Then CCS i Reduce frame by frame, expressed as follows: Among them CS i is the confidence of the emotion detected for the first time when an emotion mutation occurs, n is the number of frames from the last emotion mutation to the next emotion mutation, CS f is the confidence of the recognition result of the emotion recognition model for the fth frame, f is the frame number during the complete semantic expression, and f i is the first frame of the i-th emotion mutation, and the comprehensive emotion confidence during the entire semantic expression is: CCS is normalized, CS and CCS are both P-dimensional vectors, and the normalized comprehensive emotion confidence is similarly calculated with the second standard confidence.

6. The method according to claim 1, characterized in that The calculation method for standard confidence level includes: The standard confidence is the median of the comprehensive confidence of the statistical sample, representing the reference value of normal emotional expression. The reference value of the comprehensive expression confidence is the first standard confidence, and the reference value of the comprehensive emotion confidence is the second standard confidence. The standard confidence level can be taken as the mean of the comprehensive confidence level or the mean of the middle 50% of the distribution; Standard confidence levels are used to assess mental health issues, and different mental health issues define different standard confidence levels based on statistical results.

7. A device for evaluating real emotions, characterized in that include: An acquisition module is used to acquire the audio and video data of the test subject and separate the audio and video data into an audio stream and a video stream; A processing module, performing frame processing on the audio stream and the video stream to obtain multiple voice frames and multiple video frames; Performing face detection preprocessing on the video frame to obtain a face image, and inputting the face image into an expression recognition model to obtain expression confidence; The expression confidence is a p-dimensional vector (a1, a2, ...a p ), each vector component a i is the expression component of the expression recognition model, each expression component is a probability value predicted to be the expression, and the expression confidence is processed by the expression analysis method to obtain the expression comprehensive confidence; The expression analysis method obtains the comprehensive confidence of the expression based on the expression mutation, the audio and video frame rate FPS and the expression confidence; Inputting the speech frame into an emotion recognition model to obtain emotion confidence; The emotion confidence is a p-dimensional vector (b1, b2, ...b p ), each vector component b i is the emotion component of the emotion recognition model, each emotion component is a probability value, and the emotion confidence is processed by the emotion analysis method to obtain the emotion comprehensive confidence; The emotion analysis method obtains the comprehensive emotion confidence based on emotion mutation, audio and video frame rate FPS and the emotion confidence; The evaluation module performs similarity calculation between the comprehensive confidence of the expression and the first standard confidence, and performs similarity calculation between the comprehensive confidence of the emotion and the second standard confidence, so as to evaluate the real emotion of the test subject.

8. A video consultation system based on AI and digital human, characterized in that: The system includes, in addition to the device of claim 7, further comprising: A human-computer interaction module, including a web interface for interacting with an administrator user and a web video chat interface for interacting with a terminal user, wherein the web video chat interface collects user audio and video data through a camera and a microphone and uploads the data to a server, and the device receives the audio and video data; An AI model module, which provides the device with the required AI functions and dynamically generates audio and video data for the human-computer interaction module; Data management module, used to manage users and data processing result data; The digital human management module is used to create digital humans and maintain character attributes, images, and voice features.

9. The system according to claim 8, characterized in that The AI ​​model module includes an expression recognition model, an emotion recognition model, a speech recognition model, an LLM model, a text-generated speech model and a digital human model, the expression recognition model, the emotion recognition model, and the speech recognition model; The expression recognition model, emotion recognition model, and speech recognition model provide expression recognition, emotion recognition, and speech semantic recognition functional services to the device respectively; The LLM model automatically generates text content according to the interaction needs between the digital human and the user; The text-to-speech model generates a speech stream from the text content; The voice stream drives the digital human model to generate a video stream, and generates an audio and video stream together with the voice stream; The generated audio and video streams are presented by the human-computer interaction module.

10. The system according to claim 8, characterized in that The digital human management module includes: The digital human creation module is used by users to create digital humans, including uploading the digital human image, naming the digital human, defining the digital human role attributes, and defining the voice features; The AI ​​digital human model generates a digital human image based on a speech video or a photo of a person uploaded by a user; The naming requires that the named name uniquely identifies the digital person; The digital human role attributes refer to the digital human identity information, including one or more of occupation, gender, age, and language; The language includes Mandarin Chinese, Chinese dialects, or foreign languages; The voice features match the digital human character attributes.

Images