Video stream cognitive emotion recognition method and system based on uncertainty and complexity

Abstract

The present invention discloses a method and system for video stream cognitive emotion recognition based on uncertainty and complexity, belonging to the field of emotional computing and face recognition. S1: Obtain video information, and pre-process the video information to obtain a training image data set; S2: Construct and train a constrained local model CLM based on the training image data set, locate facial emotion description points based on the constrained local model CLM, and obtain a training data set; S3: Fuse Euclidean distance and sample entropy to extract facial emotion features in the training data set; S4: Construct a cognitive emotion recognition model based on facial emotion features and an active learning algorithm based on uncertainty and complexity and test it; S5: Obtain target video information, identify video stream cognitive emotions based on the cognitive emotion recognition model, and output the recognition result. The present invention has high efficiency, accuracy and stability.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion computing and face recognition, and more specifically to a video stream cognitive emotion recognition method and system based on uncertainty and complexity. Background Art

[0002] Human cognitive emotion recognition is an important research direction in human reliability analysis (HRA) and affective computing. The cognitive emotions of individuals during tasks largely determine the success or failure of tasks in various fields such as military, medical, and industry. It takes time for humans to convey emotions through facial expressions, so the research object of facial expression recognition (FER) should be video data. However, due to the high cost of labeling and training video samples, the feature extraction efficiency is not high, resulting in poor accuracy and real-time performance. This unfavorable factor directly or indirectly leads to a series of defects and development bottlenecks in facial expression recognition methods, including: (1) Unbalanced development, most methods are based on static images, and most video-based methods are based on deep learning methods, such as 3D ConvNet and ConvNet-RNN; (2) Lack of valuable video training datasets, and imbalanced categories in in-the-wild databases; (3) Low efficiency and low quality of video-based facial expression feature extraction.

[0003] Specifically, emotion feature extraction and emotion state recognition methods are the two main research contents and key technologies of facial expression recognition. Feature extraction includes extraction and dimensionality reduction techniques. They aim to provide more valuable and friendly inputs for classifiers. Although the methods based on convolutional neural networks (CNN) and recurrent neural networks (RNN) in deep learning can effectively extract features from static and dynamic data respectively. However, due to the lack of sufficient training data when training deep learning-based methods (deep learning-based methods require more training samples), it is easy to cause problems such as feature dimension explosion and overfitting. On the other hand, the artificially constructed facial features used by machine learning-based methods are more sensitive to the target emotions. Therefore, in many scenarios, machine learning-based methods have better recognition effects than deep learning-based methods.

[0004] Among all the machine learning-based methods, most popular facial emotion feature extraction methods extract the peak features of facial expressions from static images, such as Gabor wavelet, Local Binary Patterns (LBP), Principal Components Analysis (PCA), and Scale Invariant Feature Transform (SIFT). However, people's emotional expression is a process that takes place over time, and the temporal dynamic process of emotional expression also contains richer information. Therefore, the object of emotional feature extraction should be video data, that is, images with time series. At present, only the local binary pattern from three orthogonal planes (LBP-TOP) and its variants show good performance in dynamic FER. However, this method is relatively complex and inefficient, and cannot recognize facial expressions in real time in the task. Therefore, for the facial emotion recognition method based on machine learning, an obvious problem currently faced is the lack of a simple and effective feature extraction method based on facial dynamic information, because video data has an additional "time" dimension, which also increases the difficulty of feature description and extraction.

[0005] For sentiment classifiers based on machine learning, the most popular models are Support Vector Machine (SVM), Random Forest (RF), Probabilistic Neural Network (PNN), etc. Among them, the FER method based on SVM is the most common, but the method based on RF can often achieve the highest recognition accuracy.

[0006] In addition, the lack of machine learning-based video stream emotion recognition methods is not only due to the difficulty of feature extraction, but also due to the limitations of the dataset, which are mainly manifested in three aspects: dataset quality (the dataset contains few emotion types and the data capacity is small), data quality (the data quality is uneven, and valuable training data is drowned by other data), and sample labels (samples are not labeled, label authenticity is not verified, and label costs are too high). In other words, video-based emotion recognition methods place higher requirements on training data and usually require a large amount of more valuable training data, which means that the cost of sample collection, selection, and labeling is higher.

[0007] In the prior art, application number 202010716885.0 discloses a cognitive emotion recognition method based on geodesic distance and sample entropy, but this method does not involve the processing of emotion recognition samples, especially the labeling task of video stream samples and the cost issues brought about by the labeling difficulty, and does not involve how to improve the model accuracy in small sample data. The efficiency of locating emotion description points and feature extraction efficiency still need to be improved.

[0008] Therefore, how to quickly and effectively extract and describe facial expression features from video data, and how to select the most valuable data for model training in small sample data sets or unlabeled video data sets are problems that technical personnel in this field urgently need to solve. Summary of the invention

[0009] In view of this, the present invention provides a video stream cognitive emotion recognition method and system based on uncertainty and complexity, which are used to solve the problems existing in the above-mentioned prior art.

[0010] In order to achieve the above object, the present invention provides the following technical solutions:

[0011] On the one hand, the present invention provides a method for video stream cognitive emotion recognition based on uncertainty and complexity, comprising the following steps:

[0012] S1: Acquire video information, and preprocess the video information to obtain a training image data set;

[0013] S2: construct and train a constrained local model CLM based on the training image dataset, locate facial emotion description points based on the constrained local model CLM, and obtain a training dataset;

[0014] S3: Fusing Euclidean distance and sample entropy to extract facial emotion features in the training dataset;

[0015] S4: Based on facial emotion features and uncertainty and complexity active learning algorithm, a cognitive emotion recognition model is constructed and tested;

[0016] S5: Obtain target video information, identify the cognitive emotion of the video stream based on the cognitive emotion recognition model, and output the recognition result.

[0017] Optionally, the S1 includes the following steps:

[0018] S1.1: Get video information;

[0019] S1.2: Cut the video data of each facial expression in the video information into segments of 1-4 seconds, and form a training image dataset with several frames of images in the segments.

[0020] Optionally, S2 includes the following steps:

[0021] S2.1: Preliminary labeling of emotion description points on the training image dataset to obtain a dataset with preliminary labeling of emotion description points;

[0022] S2.2: The dataset with preliminary marked emotion description points is subjected to image alignment to obtain a training dataset after image alignment;

[0023] S2.3: performing shape modeling and local appearance modeling of the constrained local model CLM based on the image-aligned training data set to obtain a shape model and a local appearance model;

[0024] S2.4: Using the local appearance model, the shape model is corrected through the search and fitting iterative process to complete the establishment of the constrained local model. The facial emotion description points are also located during the search iterative process.

[0025] S2.5: Extract the location coordinates of the emotion description points and obtain the training data set.

[0026] Optionally, the S2.1 includes the following steps:

[0027] Assume that each video contains n images, and the emotional description point of the i-th image is expressed as:

[0028] S i =(x i,1 ,y i,1 ,x i,2 ,y i,2 ,...,x i,49 ,y i,49 )=(s i,1 ,s i,2 ,...,s i,49 );

[0029] 1≤i≤n;

[0030] Among them, s i,j =(x i,j ,y i,j ) is the two-dimensional coordinate of the jth emotion description point in the i-th image.

[0031] Optionally, S3 includes the following steps:

[0032] S3.1: Spatiotemporal feature expression of sentiment description points based on Euclidean distance, including:

[0033] For the jth emotion description point s in the i-th image i,j and the jth emotion description point s in the kth image in the same video clip k,j , the Euclidean distance between two points is:

[0034]

[0035] Extract dynamic spatiotemporal features based on the Euclidean distance between the emotion description points of each frame image of the sample;

[0036] Assuming that each video sample contains n images, the sequence of emotion description points of a sample in the training data set is expressed as:

[0037] Sample={S 1 ,S 2 ,...,S n};

[0038] The Euclidean distance between the emotion description point in the first image of each sample and the corresponding emotion description point in the subsequent images in the sequence is calculated, and the spatiotemporal feature matrix of the obtained emotion description point is as follows:

[0039]

[0040] Where, d e (V 1,j ,V i,j )=[d e (s 1,j ,s 1,j )d e (s 1,j ,s 2,j )...d e (s 1,j ,s n,j )], 1≤j≤49;

[0041] S3.2: Use sample entropy to characterize the complexity of local dynamic changes of the face when expressing cognitive emotions and obtain facial emotion features.

[0042] Optionally, the S3.2 includes:

[0043] Assume that a time series data consisting of N data is X = {x(1), x(2), ..., x(N)}. The specific sequence sample entropy calculation process is as follows:

[0044] S3.2.1: Create a sequence of m-dimensional vectors based on X {X m (1),X m (2),...,X m (N-m+1)}, where X m (i)={x(i),x(i+1),...,x(i+m-1)}, 1≤i≤N-m+1;

[0045] S3.2.2: Define the vector X m (i) and X mThe distance between (j) is the maximum absolute value of the difference between the corresponding elements in the two vectors, which is:

[0046] d{X m (i),X m (j)}=max{|x(i+t)-x(j+t)|}, 0≤t≤m-1, i≠j;

[0047] S3.2.3: For a given X m (i), 1≤i≤Nm, statistics X m (i) and X m (j) The number of distances between them that are less than or equal to r, and is recorded as B i , and define the parameters and B m (r) is:

[0048]

[0049]

[0050] S3.2.4: Increase the dimension to m+1, count X m (i) and X m (j) The number of distances between them is less than or equal to r, and is recorded as A i , define the parameters and A m (r) is:

[0051]

[0052]

[0053] S3.2.5: The sample entropy of sequence X is expressed as:

[0054] SampEn(X;m,r)=lnB m (r)-lnA m (r);

[0055] In the formula, B m (r) is the probability that any two subsequences of X match m points under similarity tolerance r, A m (r) is the probability that any two subsequences match m+1 points under similarity tolerance r;

[0056] Calculate the spatiotemporal feature matrix Feature extracted from each sample st The sample entropy of each row vector in completes the deep extraction of cognitive emotion features, and the feature vector Feature is:

[0057] Feature=SampEn(Feature st )

[0058] =SampEn{[d e (V 1,1 ,V i,1 )d e (V 1,2 ,V i,2 )...d e (V 1,49 ,V i,49 )] T},1≤i≤n

[0059] ={SampEn(1),SampEn(2),....,SampEn(49)}

[0060] Optionally, S4 includes the following steps:

[0061] S4.1: Define the unlabeled samples in the training dataset as {(x 1 ,P 1 ),(x 2 ,P 2 ),...,(x k ,P k )},x 1 to x k is the extracted feature vector Feature, P 1 To P k The probability vector corresponding to the class to which the sample belongs;

[0062] Sample x i The probability vector P of the class i The expression is:

[0063] P i =[p i,1 ,p i,2 ,p i,3 ,p i,4 ],

[0064] S4.2: Introduce information entropy to measure the uncertainty of samples and calculate any sample x in the unlabeled sample set i The uncertainty calculation formula is as follows:

[0065]

[0066] In the formula, p i,j Represented as sample x i The class probability of belonging to the jth cognitive emotion;

[0067] 4.3: Introducing complexity, the calculation formula is:

[0068] CO(xi )=|p i,1st -p i,2nd |;

[0069] In the formula, p i，1st and p i，2nd Respectively represent samples x i The probability of belonging to the largest category and the probability of the second largest category;

[0070] S4.4: Predict the class probability vector P for each sample by training the random forest model f i And generate a cognitive emotion recognition model through active learning algorithm.

[0071] Optionally, S4.4 includes constructing a decision tree according to the C4.5 algorithm, and the specific steps are:

[0072] S4.41: Divide the training data set into training data set D 1 And the training data set D u , and is the training data set D 1 Add labels to the samples in;

[0073] S4.42: Based on the training data set D 1 Train or update the random forest model f;

[0074] S4.43: For all x∈D u , use the random forest model f to calculate the class probability vector of each sample, where x refers to the samples in Du;

[0075] S4.44: For all x∈D u , calculate the sample uncertainty UN(x);

[0076] S4.45: Select training data set D u The first h samples with the largest uncertainty constitute the data set D h ;

[0077] S4.46: For all x∈D h , calculate the sample complexity CO(x);

[0078] S4.47: Pick out D h The sample x with the minimum complexity * ;

[0079] S4.48: Artificial for sample x * Add a label and change x * From D u Move to D 1 ;

[0080] S4.49: Determine whether the number of samples screened meets the requirements. If yes, execute S4.40; if not, return to S4.42;

[0081] S4.40: According to D 1 Update the random forest model f and obtain the cognitive emotion recognition model.

[0082] In one aspect, the present invention provides a video stream cognitive emotion recognition system based on uncertainty and complexity, comprising:

[0083] A preprocessing module, which obtains video information and preprocesses the video information to obtain a training image data set;

[0084] A computing module, which constructs and trains a constrained local model CLM based on a training image data set, locates facial emotion description points based on the constrained local model CLM, and obtains a training data set;

[0085] Feature extraction module, which combines Euclidean distance and sample entropy to extract facial emotion features in the training dataset;

[0086] Model building module, based on facial emotion features and uncertainty and complexity active learning algorithm to build and test cognitive emotion recognition model;

[0087] The recognition module obtains the target video information, recognizes the cognitive emotion of the video stream based on the cognitive emotion recognition model, and outputs the recognition result.

[0088] It can be seen from the above technical solutions that the present invention discloses a method and system for video stream cognitive emotion recognition based on uncertainty and complexity, and the specific beneficial effects are:

[0089] (1) The present invention has high recognition accuracy:

[0090] 1) Video-based emotion recognition: This invention uses video data to extract features and perform recognition. Since human cognitive emotion is a continuous process, video data reflecting changes in facial expressions carries more feature information. Compared with traditional emotion recognition methods based on static images, its recognition accuracy and stability will be higher;

[0091] 2) Tracking and positioning of emotion description points based on constrained local model CLM: The present invention uses CLM to track and position emotion description points. CLM is a very effective positioning model for facial emotion description points, which can accurately express facial contour features and texture features. In addition, CLM has good generalization ability and can adapt to faces of different shapes and sizes, and also has a good matching effect on slightly blurred images;

[0092] 3) The active learning algorithm based on uncertainty and complexity can screen out the most valuable samples from massive unlabeled samples. Compared with other training learning methods, under the condition of the same number of training samples, this active learning method can train a higher accuracy emotion recognition model.

[0093] (2) The present invention has high recognition efficiency and can achieve real-time recognition:

[0094] 1) Cognitive emotion spatiotemporal feature expression based on Euclidean distance and sample entropy

[0095] In step 3, each sample (a video) undergoes feature extraction processing based on Euclidean distance and sample entropy, and the data dimension of the feature vector is reduced from the original 2×49×n dimensions to 49 dimensions, significantly reducing the computational burden of the data on the classifier.

[0096] (3) This cognitive emotion recognition method can save training costs and sample labeling costs:

[0097] 1) Active learning strategy based on uncertainty and complexity

[0098] The active learning algorithm based on uncertainty and complexity proposed in the present invention can make the classification boundaries clear at a lower sample labeling cost. Different from the common active learning strategy with a single sample screening standard, this active learning strategy uses the dual standards of sample uncertainty and complexity to continuously search for the most valuable samples as training data, and uses the continuously updated training set to update and optimize the classifier. Finally, a cognitive emotion recognition model with high precision is constructed at a lower labeling cost and training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0100] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0101] Figure 2 It is the process of locating the sentiment description points based on the constrained local model CLM;

[0102] Figure 3 It is the facial emotion feature extraction process;

[0103] Figure 4 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0104] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0105] The purpose of the present invention is to provide a method and system for video stream cognitive emotion recognition based on uncertainty and complexity, the method comprising: acquiring video information, and preprocessing the video information to obtain a training image data set; constructing and training a constrained local model CLM based on the training image data set, locating facial emotion description points based on the constrained local model CLM (Constrained Local Model, CLM) to obtain a training data set; fusing Euclidean distance and sample entropy to extract facial emotion features in the training data set; constructing a cognitive emotion recognition model based on facial emotion features and an uncertainty and complexity active learning algorithm and testing it; the present invention has high efficiency, accuracy and stability.

[0106] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0107] See also Figure 1 The embodiment of the present invention discloses a method for video stream cognitive emotion recognition based on uncertainty and complexity, comprising the following steps:

[0108] S1: Acquire video information, and preprocess the video information to obtain a training image data set;

[0109] S2: construct and train a constrained local model CLM based on the training image dataset, locate facial emotion description points based on the constrained local model CLM, and obtain a training dataset;

[0110] S3: Fusing Euclidean distance and sample entropy to extract facial emotion features in the training dataset;

[0111] S4: Based on facial emotion features and uncertainty and complexity active learning algorithm, a cognitive emotion recognition model is constructed and tested;

[0112] S5: Obtain target video information, identify the cognitive emotion of the video stream based on the cognitive emotion recognition model, and output the recognition result.

[0113] The present invention first obtains the target video information, identifies the cognitive emotion of the video stream based on the cognitive emotion recognition model, and outputs the recognition result. By analyzing the relationship between emotion and cognition, a constrained local model is established to locate the facial emotion description points based on image fitting. Subsequently, facial features are extracted based on the emotion description points to achieve feature extraction, dimensionality reduction and optimization. Secondly, a sample optimization criterion and active learning strategy based on uncertainty and complexity are proposed, and an emotion recognition model based on random forest is trained based on the active learning strategy to improve the recognition accuracy at a lower training cost and sample labeling cost.

[0114] In a specific embodiment, S1 includes cognitive emotion classification and data collection:

[0115] Specifically, “doubt,” “pleasure,” “boredom,” and “frustration” were identified as the main cognitive emotions during the task.

[0116] Because the duration of cognitive emotion is about 1-4 seconds, before training the cognitive emotion recognition model, the video data of each facial expression needs to be cut into 1-4 second segments, and several frame images in the segment form the training image dataset V0.

[0117] See also Figure 2 In a specific embodiment, S2 includes locating facial emotion description points based on a constrained local model CLM:

[0118] To carry out cognitive emotion computing and recognition based on video and dynamic image data, it is first necessary to locate and track the emotion description points of the face. Only by accurately locating the emotion description points on the face can the intelligent algorithm achieve accurate emotion analysis and recognition. This method uses the Constrained Local Model (CLM) to locate and track the emotion description points. Specifically, the Constrained Local Model (CLM) model can track and locate the emotion description points, and the object of action is actually the test image or the image collected in the application scenario. The detailed steps are as follows:

[0119] S2.1: Mark emotional description points, including:

[0120] Emotional description points refer to a set of specific points that can effectively reflect changes in human facial expressions, mainly distributed in the eyes, nose tip, nostrils, mouth, mouth corners, eyebrows and facial contours. The basic principle of emotional description point marking is to use as few points as possible to describe facial expression information. In order to effectively express the characteristics of facial expressions, the present invention uses 49 emotional description points to mark facial images.

[0121] Artificially mark the emotion description points on all images in all datasets V0, and obtain the training dataset V1 with marked emotion description points. Therefore, if each video contains n images, the emotion description point of the i-th image can be expressed as:

[0122] S i =(x i,1 ,y i,1 ,x i,2 ,y i,2 ,...,x i,49 ,y i,49 )=(s i,1 ,s i,2 ,...,s i,49 ), 1≤i≤n (1)

[0123] Among them, s i,j =(x i,j ,y i,j ) is the two-dimensional coordinate of the jth emotion description point in the i-th image.

[0124] S2.2: Image alignment, including:

[0125] The training data set V1 is image aligned. Image alignment enables the model to reflect the shape changes of all training images. This step normalizes the shapes of all training images to the average shape, mainly including removing size, scaling, rotation and displacement operations to ensure that all images converge to the average shape as much as possible. The image preprocessing process is normalized using Procrustes analysis, mainly processing the rotation angle parameters, scaling parameters and two-dimensional displacement parameters of each image to obtain the training data set V2 after image alignment.

[0126] S2.3: Establish shape model and local appearance model, including:

[0127] After the image alignment is completed, the shape model and local appearance model of CLM are established based on the dataset V2.

[0128] In order to find the main law of shape vector changes of N training images, PCA is performed on all images to remove noise information and reduce data dimension, thereby obtaining a new set of mutually orthogonal variables. Finally, the shape vector of each image can be represented by a linear combination of these variables. Therefore, the shape model of the constrained local model can be expressed as:

[0129]

[0130] in, is the average shape calculated over all images in the training set. sis a set of standard orthogonal basis eigenvector matrices obtained by PCA. s is the model parameter vector.

[0131] For the local appearance model, the model consists of two parts: the local appearance information extraction process and the local classifier. The local appearance model uses the local binary patterns operator (LBP) to extract local appearance information, and uses the support vector machine (SVM) as the local classifier for subsequent shape fitting;

[0132] S2.4: CLM model generation and facial emotion description point positioning based on CLM image fitting, including:

[0133] The local appearance model is used to search near the feature points to determine the position of each feature point. The shape model is automatically corrected through the search and fitting iterative process. While establishing the constrained local model, the facial emotion description points are also located.

[0134] The search and fitting iterative process is as follows:

[0135] Input: initial shape model, the coordinates of the emotion description points of each image in the training set V2, and the test image;

[0136] S2.41: For each emotion description point, extract the surrounding LBP feature vector and use the SVM local classifier to search around the feature point, calculate the corresponding response value (matching degree), and thus obtain the response surface around each feature point. For the test image, the emotion description point will be initially assigned based on the shape model;

[0137] S2.42: Fit the current shape according to the current shape model, and update the corresponding shape model parameters and the position of the emotion description point for the next iteration;

[0138] S2.43: Determine whether the shape model converges. If the parameter change of the shape model is less than a certain threshold, the model converges, otherwise jump to S2.41; for the test image, it is to determine whether the position change of the emotion description point is less than a certain threshold. If so, it converges, otherwise jump to S2.41.

[0139] Output: The fitted shape model and the test image after the emotion description points are located.

[0140] S2.5: Extract the location coordinates of the emotion description points:

[0141] The purpose of fitting and establishing the constrained local model CLM is to track and locate the emotion description points in the test data or real-time images, so as to pave the way for extracting more accurate and effective feature vectors. Therefore, for the training data set V2 that has already manually marked the emotion description points in advance, the emotion description point coordinate set of each image in each video can be directly extracted and arranged into a sequence in time sequence, which is recorded as data set V3; for the test data or real-time data, the emotion description point coordinate set of each image after CLM positioning should be extracted before cognitive emotion recognition and arranged into a sequence in time sequence, which is recorded as data set V4. A sequence extracted from each video is called a sample. The data format of the emotion description point coordinate set in each sequence (sample) in V3 and V4 is shown in formula (1).

[0142] The constrained local model CLM is used to locate emotion description points. Compared with the active appearance model used in the existing technology, which establishes a global texture change model during the training process, CLM only establishes a texture template around the emotion description point. During the search process, it is only necessary to compare the texture information of the local emotion description point and use the downhill simplex method to find the offset of the emotion description point, which is more efficient and has higher accuracy and stability.

[0143] In a specific embodiment, S3: extracting facial emotion features by integrating Euclidean distance and sample entropy, comprises:

[0144] The emotion description points obtained by the constrained local model (CLM) only express the peak value of facial emotions in static images, while the expression of cognitive emotions is a process of duration and cannot be described by a single peak image. Therefore, this method proposes a facial emotion feature extraction method for video data based on Euclidean distance and sample entropy.

[0145] S3.1: Spatiotemporal feature expression of sentiment description points based on Euclidean distance:

[0146] The spatial features of emotional description points are usually expressed by measuring distance. For the jth emotional description point s in the i-th image i,j and the jth emotion description point s in the kth image in the same video clip k,j , the Euclidean distance between two points is:

[0147]

[0148] Then, dynamic spatiotemporal features are extracted based on the Euclidean distance of the emotion description points between each frame of the sample image.

[0149] Assuming that each video sample contains n images, the sequence of emotion description points of a sample in dataset V3 or V4 can be expressed as:

[0150] Sample={S 1,S 2 ,...,S n} (4)

[0151] At this time, the data dimension of a sample is 2×49×n.

[0152] The Euclidean distance between the emotion description point in the first image of each sample and the corresponding emotion description point in the subsequent images in the sequence is calculated, and the spatiotemporal feature matrix of the obtained emotion description point is as follows:

[0153]

[0154] Among them, d e (V 1,j ,V i,j )=[d e (s 1,j ,s 1,j )d e (s 1,j ,s 2,j )...d e (s 1,j ,s n,j )], 1≤j≤49. At this time, the data dimension of each sample becomes 49×n.

[0155] S3.2: Facial emotion feature description based on sample entropy:

[0156] In order to highlight the characteristic patterns of facial emotion features and provide more friendly input for the classifier, the facial emotion features are further optimized by extracting sample entropy. Sample entropy does not depend on the quality of the data, and only a small amount of data is needed to accurately calculate the complexity and regularity of the sample. Therefore, the present invention uses sample entropy to characterize the complexity of local dynamic changes in the face when expressing cognitive emotions.

[0157] Assume that a time series data consisting of N data is X = {x(1), x(2), ..., x(N)}, the calculation process of the sequence sample entropy is as follows:

[0158] Step 3.2.1: Create a sequence of m-dimensional vectors based on X {X m (1),X m (2),...,X m (N-m+1)},

[0159] Where X m (i)={x(i),x(i+1),...,x(i+m-1)}, 1≤i≤N-m+1;

[0160] Step 3.2.2: Define vector X m (i) and X m(j) The distance between them is the difference between the corresponding elements in the two vectors

[0161] The maximum absolute value, that is:

[0162] d{X m (i),X m (j)}=max{|x(i+t)-x(j+t)|}, 0≤t≤m-1, i≠j (6)

[0163] Step 3.2.3: For a given X m (i), 1≤i≤Nm, statistics X m (i) and X m (j) Small distance between

[0164] equal to r, and is recorded as B i , and define the parameters and B m (r) is:

[0165]

[0166]

[0167] Step 3.2.4: Increase the dimension to m+1 and count X in the same way m (i) and X m (j) The number of distances between them is less than or equal to r, and is recorded as A i , and define the parameters and A m (r) is:

[0168]

[0169]

[0170] Step 3.2.5: The sample entropy of sequence X can be expressed as:

[0171] SampEn(X;m,r)=lnB m (r)-lnA m (r) (11)

[0172] Among them, B m (r) is the probability that any two subsequences of X match m points under similarity tolerance r, A m (r) is the probability that any two subsequences match m+1 points with a similarity tolerance of r.

[0173] The parameter is set to m = 2 and r is 0.2 times the standard deviation of the original signal. According to the above steps 3.2.1 to 3.2.5, the spatiotemporal feature matrix Feature extracted from each sample is calculatedst The sample entropy of each row vector in is used to complete the deep extraction of cognitive emotion features. The feature vector Feature is shown as follows:

[0174]

[0175] After completing the facial emotion feature extraction based on sample entropy, the data dimension of each sample is reduced to 49 dimensions. A feature vector based on Euclidean distance and sample entropy is extracted for each sample in the data sets V3 and V4, which are respectively recorded as V5 (V5 is used to subsequently establish an active learning model and train a classifier) ​​and V6 (V6 can be used to test the performance of this cognitive emotion recognition method).

[0176] At this point, the facial emotion feature extraction is completed by integrating Euclidean distance and sample entropy, as shown in Figure 3 As shown in FIG. 1 , the process of locating emotion description points and extracting features for a piece of video data is performed.

[0177] In a specific embodiment, S4: constructing a cognitive emotion recognition model based on active learning of uncertainty and complexity, comprising:

[0178] Most emotion recognition methods are supervised learning methods, which not only require a large number of samples, but also the samples usually do not have enough labels. Furthermore, unlike image-based emotion recognition, each video cognitive emotion sample is composed of a series of images, and the amount of data is huge. At the same time, considering the complex structure of the characteristic data of the cognitive emotion time series space, the video-based emotion recognition model requires more high-quality samples to support it. In addition, unlabeled data is usually easy to obtain, while manual labeling is time-consuming and labor-intensive, and consumes a lot of costs. Labeling training samples not only requires the participation of a large number of experts, but also takes a lot of time. This seriously affects the training efficiency and feasibility of video-based emotion recognition methods. Therefore, the present invention proposes a classifier learning method that can make full use of small sample labeled data and unlabeled data to achieve efficient and accurate cognitive emotion recognition, that is, an active learning method based on uncertainty and complexity.

[0179] Active learning is a machine learning strategy that actively selects the most valuable data and queries its labels, thereby reducing label costs and improving classifier accuracy. The key to active learning is the sample selection criterion. However, most active learning algorithms only deploy one selection criterion, which may limit their performance. This paper proposes a dual-criteria active learning method based on uncertainty and complexity. Uncertainty is used to describe the confusion of samples, and complexity is used to measure the difference between local samples and global samples. The following is the definition of sample uncertainty and complexity and the preparation for training cognitive emotion recognition models.

[0180] Define the unlabeled sample as {(x1 ,P 1 ),(x 2 ,P 2 ),...,(x k ,P k )},x 1 to x k is the extracted feature vector Feature, P 1 To P k The probability vector corresponding to the class to which the sample belongs, any sample x in the above unlabeled sample set i That is the feature vector Feature extracted in step 3, P i For sample x i The probability vector of the possible class. Since this method defines four cognitive emotions, P i The expression is as follows:

[0181]

[0182] To obtain the class probability vector P i As well as making the cognitive emotion recognition model have a higher training speed and classification speed, reducing the risk of overfitting, training the random forest model to predict the class probability vector P of each sample i The cognitive emotion recognition model is generated through active learning algorithm. The proportion of the classification results of each decision tree in the random forest to the total number of samples is used as the probability that the sample belongs to each cognitive emotion. The decision tree is constructed according to the C4.5 modeling rules.

[0183] 1) Uncertainty

[0184] Samples with greater uncertainty often contain more information and are difficult to classify. In order to measure uncertainty, information entropy is introduced to measure the uncertainty of samples. Sample x i The uncertainty calculation formula is as follows:

[0185]

[0186] In the formula, p i,j Represented as sample x i Class probability of belonging to the jth cognitive emotion.

[0187] Uncertainty is used to help active learning algorithms find some samples that are difficult to classify.

[0188] 2) Complexity

[0189] For multi-label classification, the correct labels tend to be concentrated in the top two categories according to the label probability sorting. This concentration of labels in the top two is what causes complexity and makes it difficult to classify. Therefore, this paper introduces complexity and defines it as the distance from the sample with the highest probability to the sample with the second highest probability. Its calculation formula is:

[0190] CO(x i )=|p i,1st -p i,2nd | (15)

[0191] where p i，1st and p i，2nd Respectively represent x i The probability of the largest class and the probability of the second largest class. Complexity is used to select the most difficult sample to classify in this round of learning.

[0192] At this point, the cognitive emotion recognition model can be constructed using an active learning framework based on uncertainty and complexity.

[0193] Step 4.1: Divide the dataset V5 (the default unlabeled dataset) into dataset D 1 and D u , and D 1 The samples in are manually labeled.

[0194] Specifically, we first use a small number of labeled data sets D 1 After rounds of iterations, the unlabeled samples in Du are gradually labeled one by one and added to D 1 In the implementation of the labeled dataset D 1 The expansion of the dataset D 1 and D u The specific ratio depends on the actual situation.

[0195] Step 4.2: According to D 1 Train or update the random forest model f.

[0196] Step 4.3: For all x∈D u , use model f to calculate the class probability vector of each sample, x refers to the samples in Du.

[0197] Step 4.4: For all x∈D u , calculate the sample uncertainty UN(x).

[0198] Step 4.5: Pick out D u The first h samples with the largest uncertainty in constitute D h .

[0199] Step 4.6: For all x∈D h, calculate the sample complexity CO(x).

[0200] Step 4.7: Pick out D h The sample x with the minimum complexity * .

[0201] Step 4.8: Artificial sample x * Add a label and change x * From D u Move to D 1 .

[0202] Step 4.9: Determine whether the number of samples screened meets the requirements. If so, execute step 4.10; if not, return to step 4.2.

[0203] Step 4.10: According to D 1 Update the random forest model f and obtain the cognitive emotion recognition model.

[0204] See also Figure 4 Based on the above method steps, a cognitive emotion recognition system based on uncertainty and complexity video stream was developed on the PC side, which can recognize cognitive emotions in real time, including:

[0205] A preprocessing module, which obtains video information and preprocesses the video information to obtain a training image data set;

[0206] Specifically, real-time facial expression video data collected by the camera is obtained, and the video stream is cut into video clips of 1 second in length automatically or after each click of the "recognition" button, and input into the feature extraction module.

[0207] A computing module, which constructs and trains a constrained local model CLM based on a training image data set, locates facial emotion description points based on the constrained local model CLM, and obtains a training data set;

[0208] Feature extraction module, which combines Euclidean distance and sample entropy to extract facial emotion features in the training dataset;

[0209] Specifically, the emotion description points of the face in the video are tracked and located, and the feature vector based on Euclidean distance and sample entropy is extracted from the video and input into the emotion recognition module.

[0210] Model building module, based on facial emotion features and uncertainty and complexity active learning algorithm to build and test cognitive emotion recognition model;

[0211] The recognition module obtains the target video information, recognizes the cognitive emotion of the video stream based on the cognitive emotion recognition model, and outputs the recognition result.

[0212] Specifically, the recognition module deploys a random forest-based cognitive emotion recognition model trained by the proposed active learning strategy to achieve recognition and classification of cognitive emotions.

[0213] In a specific embodiment, it also includes a cognitive emotional state output and reminder module to obtain the currently recognized cognitive emotional state and display it in real time on the interface in a probabilistic manner. When the recognition result is doubt, boredom and frustration and lasts for a period of time, an alarm will be issued to remind the executor to terminate the current task or seek help.

[0214] As for the system device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0215] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0216] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video stream cognitive emotion recognition method based on uncertainty and complexity, characterized in that: The following steps are involved: S1: Acquire video information, and preprocess the video information to obtain a training image data set; S2: construct and train a constrained local model CLM based on the training image dataset, locate facial emotion description points based on the constrained local model CLM, and obtain a training dataset; S3: Fusing Euclidean distance and sample entropy to extract facial emotion features from the training dataset, including: S3.1: Spatiotemporal feature expression of sentiment description points based on Euclidean distance: The spatial characteristics of emotional description points are usually expressed by measuring distance; for the jth emotional description point s in the i-th image i,j and the jth emotion description point s in the kth image in the same video clip k,j , the Euclidean distance between two points is: Assuming that each video sample contains n images, the sequence of emotion description points of a sample in dataset V3 or V4 can be expressed as: Sample={S1,S2,...,S n } (4) The Euclidean distance between the emotion description point in the first image of each sample and the corresponding emotion description point in the subsequent images in the sequence is calculated, and the spatiotemporal feature matrix of the obtained emotion description point is as follows: Among them, d e (V 1,j ,V i,j )=[d e (s 1,j ,s 1,j )d e (s 1,j ,s 2,j )...d e (s 1,j ,s n,j )], 1≤j≤49; at this time, the data dimension of each sample becomes 49×n; Step 3.2.5: The sample entropy of sequence X can be expressed as: SampEn(Xm,r)lnB m (r)-lnA m (r) (11) Among them, B m (r) is the probability that any two subsequences of X match m points under similarity tolerance r, A m (r) is the probability that any two subsequences match m+1 points under similarity tolerance r; The parameter is set to m = 2 and r is 0.2 times the standard deviation of the original signal; calculate the spatiotemporal feature matrix Feature extracted from each sample st The sample entropy of each row vector in is used to complete the deep extraction of cognitive emotion features; the feature vector Feature is as follows: S4: Based on facial emotion features and uncertainty and complexity active learning algorithm, a cognitive emotion recognition model is constructed and tested, including the following steps: S4.1: Define the unlabeled samples in the training dataset as {(x1,P1),(x2,P2),...,(x k ,P k )}, x1 to x k is the extracted feature vector Feature, P1 to P k The probability vector corresponding to the class to which the sample belongs; S4.2: Introduce information entropy to measure the uncertainty of samples and calculate any sample x in the unlabeled sample set i uncertainty; S4.3: Calculate sample x i complexity; S4.4: Predict the class probability vector of each sample by training the random forest model f, and generate a cognitive emotion recognition model through an active learning algorithm; S5: Obtain target video information, identify the cognitive emotion of the video stream based on the cognitive emotion recognition model, and output the recognition result.

2. According to the uncertainty and complexity based video stream cognitive emotion recognition method of claim 1, it is characterized in that: The S1 comprises the following steps: S1.1: Get video information; S1.2: Cut the video data of each facial expression in the video information into segments, and form a training image dataset with several frame images in the segments.

3. The method for video stream cognitive emotion recognition based on uncertainty and complexity according to claim 1, characterized in that: The S2 comprises the following steps: S2.1: Preliminary labeling of emotion description points on the training image dataset to obtain a dataset with preliminary labeling of emotion description points; S2.2: The dataset with preliminary marked emotion description points is subjected to image alignment to obtain a training dataset after image alignment; S2.3: performing shape modeling and local appearance modeling of the constrained local model CLM based on the image-aligned training data set to obtain a shape model and a local appearance model; S2.4: Based on the shape model and the training image dataset, the facial emotion description points are located through search and fitting iterations; S2.5: Extract the location coordinates of the emotion description points and obtain the training data set.

4. The method for video stream cognitive emotion recognition based on uncertainty and complexity according to claim 3 is characterized in that: The S2.1 comprises the following steps: Each video is assumed to contain n images, and the emotional description point of the i-th image is expressed as: S i =(x i,1 ,y i,1 ,x i,2 ,y i,2 ,...,x i,49 ,y i,49 )=(s i,1 ,s i,2 ,...,s i,49 ); 1≤i≤n; Among them, s i,j =(x i,j ,y i,j ) is the two-dimensional coordinate of the jth emotion description point in the i-th image.

5. The video stream cognitive emotion recognition method based on uncertainty and complexity according to claim 1 is characterized in that: The sample x in S4.1 i The probability vector P of the class i The expression is: In the formula, p i,1 、p i,2 、p i,3 、p i,4 The samples x are i Class probabilities belonging to 4 different kinds of cognitive emotions.

6. The method for video stream cognitive emotion recognition based on uncertainty and complexity according to claim 5, characterized in that: S4.2 enters the information entropy to measure the uncertainty of the sample, sample x i The uncertainty calculation formula is as follows: In the formula, p i,j Represented as sample x i Class probability of belonging to the jth cognitive emotion.

7. The method for video stream cognitive emotion recognition based on uncertainty and complexity according to claim 5, characterized in that: S4.3: Sample x i The complexity calculation formula is: WHAT(x i )=|p i,1st -p i,2nd |; In the formula, p i，1st and p i，2nd Respectively represent samples x i The probability of belonging to the largest class and the probability of the second largest class.

8. The method for video stream cognitive emotion recognition based on uncertainty and complexity according to claim 1, characterized in that: S4.4 includes constructing a decision tree according to the C4.5 algorithm, and the specific steps are: S4.41: Divide the training data set into training data set D1 and training data set D u , and add labels to the samples in the training dataset D1; S4.42: Train or update the random forest model f based on the training data set D1; S4.43: For all x∈D u , use the random forest model f to calculate the class probability vector of each sample; S4.44: For all x∈D u , calculate the sample uncertainty UN(x); S4.45: Select training data set D u The first h samples with the largest uncertainty constitute the data set D h ; S4.46: For all x∈D h , calculate the sample complexity CO(x); S4.47: Pick out D h The sample x with the minimum complexity * ; S4.48: Artificial for sample x * Add a label and change x * From D u Move to D1; S4.49: Determine whether the number of samples screened meets the requirements. If yes, execute S4.40; if not, return to S4.42; S4.40: Update the random forest model f according to D1 to obtain the cognitive emotion recognition model.

9. A system using the video stream cognitive emotion recognition method based on uncertainty and complexity according to any one of claims 1 to 8, characterized in that: include: A preprocessing module, which obtains video information and preprocesses the video information to obtain a training image data set; A computing module, which constructs and trains a constrained local model CLM based on a training image data set, locates facial emotion description points based on the constrained local model CLM, and obtains a training data set; Feature extraction module, which combines Euclidean distance and sample entropy to extract facial emotion features in the training dataset; Model building module, based on facial emotion features and uncertainty and complexity active learning algorithm to build and test cognitive emotion recognition model; The recognition module obtains the target video information, recognizes the cognitive emotion of the video stream based on the cognitive emotion recognition model, and outputs the recognition result.

Images