Group facial expression and emotion recognition method and system based on spatiotemporal consistency

Abstract

The present disclosure proposes a method and system for group facial expression and emotion recognition based on spatiotemporal consistency, comprising the following steps: performing face detection and recognition on a video image of a crowd motion scene to be recognized to obtain the facial expression of each individual in each frame; identifying whether the individual's emotion is panic based on the facial expression of each individual; calculating whether the occluded individuals in some frame images are panic-emotional based on temporal consistency, and calculating whether the occluded individuals in all frame images are panic-emotional based on spatial consistency; and obtaining the group panic degree based on the proportion of panic-emotional individuals among all individuals. The present disclosure solves the problem of incomplete facial expression information collection due to occlusion in a crowd, based on the proposed temporal consistency individual emotion calculation method and spatial consistency individual emotion calculation method, and then solves the problem of group emotion recognition in crowd motion scenes through the group panic degree.

Description

Technical Field

[0001] The present disclosure relates to the technical fields related to deep learning and computer vision technology, and more specifically, to a method and system for group facial expression and emotion recognition based on spatiotemporal consistency. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In recent years, research on crowd motion simulation has gained significant attention. With the rapid development of urbanization and improved living standards, people are increasingly frequenting public places, resulting in high crowd density and mobility in places like train stations, schools, and large entertainment and shopping malls. Dangerous situations, such as fires and earthquakes, can cause panic among the crowd, leading to congestion, delays, stampedes, and other accidents. Identifying the true emotions of the crowd and then implementing appropriate emotional control measures to reduce panic is an effective way to ensure safe evacuations.

[0004] In crowd-moving scenes, identifying the emotions of each individual through facial expression recognition, defining the group's panic level using statistical methods, and further determining the group's collective emotions are important issues to address. In crowd-moving scenes, occlusions can occur due to the high mobility of the crowd, resulting in incomplete facial expression information collection. Therefore, identifying the emotions of obscured individuals, which results in incomplete information, poses a challenge. The inventors discovered that traditional group emotion recognition methods cannot identify the emotions of people in crowd-moving scenes, nor do they address the problem of facial expression recognition when facial occlusions result in incomplete information. Summary of the Invention

[0005] In order to solve the above problems, the present disclosure proposes a method and system for group facial expression and emotion recognition based on spatiotemporal consistency. Based on the proposed temporal consistency individual emotion calculation method and spatial consistency individual emotion calculation method, the problem of incomplete facial expression information collection caused by occlusion in the crowd is solved. The group panic degree is used to solve the problem of group emotion recognition in crowd motion scenes. The method can identify the panic degree of a group, thereby providing real data for the study of crowd motion.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] One or more embodiments provide a method for group facial expression and emotion recognition based on spatiotemporal consistency, comprising the following steps:

[0008] Perform face detection and recognition on the acquired video images of the crowd motion scene to be identified to obtain the facial expression of each individual in each frame of the image;

[0009] Identify whether the individual's emotion is panic based on each individual's facial expression; calculate whether the individual occluded in some frame images is panic based on temporal consistency, and calculate whether the individual occluded in all frame images is panic based on spatial consistency;

[0010] The group panic degree is obtained based on the proportion of panic-stricken individuals among all individuals.

[0011] One or more embodiments provide a group facial expression and emotion recognition system based on spatiotemporal consistency, including:

[0012] Face recognition module: configured to perform face detection and recognition on the acquired video images of the crowd motion scene to be recognized, and obtain the facial expression of each individual in each frame of the image;

[0013] Individual panic judgment module: configured to identify whether the individual's emotion is panic based on the facial expression of each individual; calculate whether the individual occluded in some frame images is panic based on temporal consistency, and calculate whether the individual occluded in all frame images is panic based on spatial consistency;

[0014] The group panic degree judgment module is configured to obtain the group panic degree according to the proportion of panicking individuals among all individuals.

[0015] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above method are completed.

[0016] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are completed.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] This method, based on temporally consistent and spatially consistent individual emotion calculation, addresses the issue of incomplete facial expression information due to crowd occlusion. It can identify whether all individuals in a video are in a panic state, while also quantifying the degree of panic within the group. This method can accurately capture group emotions and identify the degree of panic even when facial expression information is incomplete due to crowd occlusion. This provides accurate data support for crowd movement research, such as crowd evacuation. It boasts a high degree of simplification, modularity, high model accuracy, and accurate adaptation to the scenario.

[0019] The advantages of the present disclosure and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0021] Figure 1 is a flow chart of the method of embodiment 1 of the present disclosure;

[0022] Figure 2 It is a schematic diagram of the residual module structure of Example 1 of the present disclosure. DETAILED DESCRIPTION

[0023] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0025] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0026] Example 1

[0027] In the technical solutions disclosed in one or more embodiments, Figure 1 As shown in FIG, the group expression emotion recognition method based on spatiotemporal consistency includes the following steps:

[0028] Step 1: Perform face detection and recognition on the acquired video image of the crowd motion scene to be identified to obtain the facial expression of each individual in each frame of the image;

[0029] Step 2: Identify whether the individual's emotion is panic based on the facial expression of each individual; calculate whether the individual occluded in some frame images is panic based on temporal consistency, and calculate whether the individual occluded in all frame images is panic based on spatial consistency;

[0030] Identify whether the individual's emotion is panic based on the facial expression of each individual. Optionally, input the facial expression of each individual in each frame of the image into a trained Resnet34 network, and output whether the emotion of the unobstructed individual is panic.

[0031] Step 3: Calculate the group panic level based on the proportion of panicking individuals among all individuals. The optimal crowd evacuation plan can be generated based on the group panic level.

[0032] The crowd evacuation plan includes evacuation route planning, crowd evacuation exit selection, etc.

[0033] In this embodiment, a method for calculating individual emotions based on temporal consistency and spatial consistency addresses the issue of incomplete facial expression information due to crowd occlusion. This method can identify whether all individuals in a video are experiencing panic, while also quantifying the degree of panic within the group. This method can accurately capture group emotions and identify the degree of panic even when facial expression information is incomplete due to crowd occlusion. This method provides accurate data support for crowd movement research, such as crowd evacuation. It boasts a high degree of simplification, modularity, high model accuracy, and accurate adaptation to the scenario.

[0034] In step 1, the method of performing face detection and recognition on the acquired video image of the crowd motion scene to be recognized to obtain the facial expression of each individual in each frame of the image includes the following steps:

[0035] Step (1) Input the RGB video of the crowd motion scene to be identified.

[0036] Step (2) divides the video into frames and saves them.

[0037] Step (3) performs face detection on each frame of the image processed by frame division.

[0038] Step (4) crops and saves the faces detected in each image.

[0039] Step (5) inputs the cropped face image into the deep residual network (Resnet34) structure to obtain the facial expression recognition result.

[0040] In the step (1), an RGB video of a crowd motion scene to be identified is obtained;

[0041] Optionally, the cap=cv.VideoCapture() function may be used to input a video to be recognized. The video to be recognized may be an RGB video that can capture facial expressions of a crowd in a scene of crowd movement.

[0042] In the step (2), the video is divided into frames to obtain and save pictures frame by frame;

[0043] Specifically, you can use the face_detect_demo(frame) function to divide the input video into frames, obtain each frame image and save it in a folder.

[0044] In the step (3), face detection is performed on each frame of the image processed by frame division;

[0045] Optionally, the API interface of OpenCV can be called to detect faces in the image. Specifically, the face_detect=cv.CascadeClassifier function is used to call the "haarcascade_frontalface_default.xml" face detection module in OpenCV.

[0046] Optionally, the face detection algorithm can be implemented using a Haar-based feature extraction algorithm.

[0047] The calculation characteristic of the Haar algorithm is to subtract the white value from the black value in an area of ​​an image. In this embodiment, an integral image can be used as a method to accelerate the calculation. The integral image of coordinate A(x,y) is the sum of all pixels in its upper left corner. The integral image is defined as follows:

[0048]

[0049] Where I(x,y) represents the integral image, and i(x,y) represents the original image. Each point (x,y) in the integral image is the sum of all values ​​in the upper left corner of the region corresponding to that point, and the integral image can be obtained by traversing the image only once. For any point (x,y), the integral image is calculated as follows:

[0050] I(x,y)=G(x,y)+I(x-1,y)+I(x,y-1)-I(x-1,y-1) (2)

[0051] Where I represents the integral value of the point (x, y), and G represents the grayscale value of the point (x, y).

[0052] In this embodiment, the integral image can be used to efficiently calculate the feature area in the image.

[0053] In the step (4), the faces detected in each picture are cropped to obtain pictures containing only faces, and the cropped pictures are saved.

[0054] Crop the detected face to get an image containing only the face. Optionally, you can use the cv2.rectangle() function to set a rectangular box to select the detected face. Use the cv2.imwrite() function to crop the selected face.

[0055] The steps to crop the detected face to obtain a picture containing only the face can be as follows:

[0056] (4-1) For each image after face detection, set a rectangular frame close to the size of the face to select the detected face.

[0057] Specifically, the image after face detection is converted into grayscale, the grayscale image is histogram-normalized, the face area is detected in the obtained image matrix, and then the face area is framed with a rectangular frame.

[0058] (4-2) Cut out the faces selected in each image and label them, then output each face in the image in the order of the labels.

[0059] In the step (5), the cropped face image is input into the deep residual network structure to obtain the recognition result of the facial expression.

[0060] Among them, the deep residual network can be a Resnet34 network.

[0061] In this embodiment, the trained Resnet34 network structure is used to accurately recognize the emotional results of facial expressions.

[0062] Optionally, the Resnet34 network structure of this embodiment includes a convolutional layer, a residual module, a pooling layer, a fully connected layer and a Softmax classifier, and the convolutional layer, the residual module, the pooling layer, the fully connected layer and the Softmax classifier are connected in sequence.

[0063] Specifically, a face image is selected as input and put into the trained Resnet34 network structure. After the convolution layer, pooling layer and fully connected layer, the quantitative value and label of the individual emotion are finally output.

[0064] The convolutional layer is used to extract image features and information from the input image.

[0065] Among them, image features are reflected by each pixel in the image in a combined or independent manner, such as the texture features and color features of the image.

[0066] The pooling layer is used to select features extracted by the convolutional layer. Optionally, the pooling operation can be max pooling or average pooling. The pooling layer calculates the maximum and average values ​​of the n*n matrix by sliding the n*n matrix window.

[0067] The fully connected layer is used to convert all feature matrices of the pooling layer into a one-dimensional feature vector. The fully connected layer is placed at the end of the Resnet34 network structure to classify the image into the fully connected layer and then output the result.

[0068] For facial expression recognition, traditional methods mainly rely on the pros and cons of manually extracted features, and the algorithm has poor robustness. Traditional convolutional neural networks cannot extract deeper image features. Therefore, this embodiment will use a deep residual network for facial expression recognition.

[0069] The ResNet network is mainly composed of residual modules. The output of the residual module is sent to the fully connected layer for feature fusion, and finally classified by the Softmax classifier.

[0070] Deep convolutional networks naturally integrate features from different levels, from low to medium to high. The feature hierarchy can be enriched by deepening the network. Therefore, when building a convolutional network, the deeper the network, the richer the level of features that can be extracted. Therefore, deeper network structures are generally preferred to obtain higher-level features. However, when using deep network structures, problems such as vanishing gradients, exploding gradients, and network degradation may arise. Compared to traditional convolutional neural networks, deep residual networks introduce residual modules into the network. This module effectively alleviates the vanishing gradient problem in backpropagation during network model training, thereby addressing the difficulty of training deep networks and the performance degradation they suffer from.

[0071] The Softmax classifier is often used in multi-classification recognition. Its input is a vector of arbitrary real numbers, and its output is a vector where each element has a value between (0, 1) and its sum is 1. Suppose there is an array whose Softmax expression is:

[0072]

[0073] Among them, S i Is the value of Softmax, e i represents the i-th element, ∑ j e j Represents the sum of all elements.

[0074] Figure 2The basic unit of a residual network is the residual module. You can see that x is the input of this residual block, also called F(x) the residual. x is the input value, and F(x) is the output after the first layer undergoes linear transformation and activation. The diagram shows that in a residual network, after the second layer undergoes linear transformation and before activation, F(x) is added to the input value x of this layer, and then activated and output. x is added before the second layer's output is activated. The initial input x is weighted and then passed through the activation function, as in a conventional neural network. After this weighted addition, the input signal and the output are then superimposed and passed through the activation function. A typical example is a line that connects directly to the output. This path is called a shortcut. The residual in a linear fit is the difference between the function value of a data point and the fitted line. Here, x is the fitted function, and H(x) is the specific data point. Through training, the fitted value plus F(x) yields the value of the specific data point. Therefore, F(x) represents the residual.

[0075] Since the images are generated by framing a group video, one image will contain multiple individual faces. They are input into the Resnet34 network structure in the order of the previous labels to obtain the true emotions reflected by the facial expressions of each individual in the image.

[0076] Furthermore, the network training steps are also included, as follows:

[0077] (5-1) Obtain facial expression datasets as training and test sets;

[0078] In this example, the ResNet34 network architecture was trained and tested using the RAFDB facial expression dataset. This dataset consists of original images and aligned facial images. The aligned images contain 12,271 training samples and 3,068 aligned test samples. This dataset was used for pre-training, resulting in a 1,500 epoch pre-trained model. This model achieved an accuracy of 74% on the test set.

[0079] (5-2) The images in the dataset are used as input, and the individual emotions are used as output, which are input into the Resnet34 network structure for training to obtain the trained Resnet34 network structure.

[0080] Furthermore, based on the trained Resnet34 network structure, the cropped face images to be detected are recognized, and the results of individual emotion recognition are expressed in the form of a matrix to obtain the emotion recognition result matrix R.

[0081] Optionally, the above individual emotion recognition results can be expressed in the form of a matrix. Assuming that a video is divided into m frames and there are n individuals in the group in the video, the emotion recognition result matrix R is as follows:

[0082]

[0083] [r i1 …r ij …r in ] (5)

[0084]

[0085] Wherein, formula (5) represents the emotion recognition result of each individual in the group at the moment of frame i. Formula (6) represents the emotion recognition result of individual j in all frames of the video. ij Represents the emotion recognition result of individual j at the moment of frame i, and defines r ij Possible values ​​are as follows:

[0086]

[0087] Among them, the emotion recognition result of individual j at the moment of frame i is non-panic emotion, which is recorded as r ij =1; if the emotion recognition result of individual j at the moment of frame i is panic, it is recorded as r ij = -1; if no facial expression is detected for individual j at the moment of frame i, it is recorded as r ij =0.

[0088] In step 2, the Resnet34 network can also be used to identify whether the emotion of the unobstructed individual is panic. The facial expression of each individual in each frame of the image is input into the trained Resnet34 network, and the output is whether the emotion of the unobstructed individual is panic.

[0089] The training process includes: constructing a training set including facial expressions and corresponding emotions, taking each individual's facial expression as input and individual emotions as output training, and obtaining a trained Resnet34 network.

[0090] In step 2, based on the temporal consistency of individual emotion calculation and the spatial consistency of individual emotion calculation, the emotion of the occluded individual is calculated.

[0091] In crowd movement scenes, the negative emotion of panic is the most instructive, so panic recognition is of great significance. In real crowd movement scenes, due to the movement and density of the crowd, some individuals will have their facial expressions obscured at one or more different time periods, resulting in the inability to capture their facial expressions for a certain frame or frames. Alternatively, some individuals will have their facial expressions obscured for the entire time period, resulting in the inability to capture their facial expressions for any frame. To this end, this embodiment introduces the concepts of calculating temporally consistent individual emotions and spatially consistent individual emotions to assist in deriving an individual's emotions when facial information is incomplete or not captured.

[0092] Individual emotion calculation based on temporal consistency means that the emotions reflected by facial expressions in different frames will be different, and it solves the problem of individual emotion recognition when the facial expression of an individual A cannot be collected in a certain frame or several frames due to occlusion.

[0093] In some embodiments, optionally, a method for calculating whether an occluded individual in a partial frame image is in a panic emotion based on temporal consistency is as follows: among all frame images that can capture the complete facial expression information of individual a, the proportion of frames identified as panic emotions is greater than a set proportion threshold, that is, the emotion of individual a is determined to be panic in the video of the identified crowd movement, otherwise it is determined to be non-panic.

[0094] Optionally, the ratio threshold is set to be no less than 50%.

[0095] Specifically, in this embodiment, if the number of frames in which individual a can collect complete facial expression information and is identified as panic emotion accounts for more than half of the number of frames in which individual a can collect complete facial expression information, it can be determined that the individual is in panic emotion in the video of crowd movement.

[0096] In this embodiment, the recognition of individual facial expressions and emotions based on temporal consistency solves the problem that an individual's facial expression may be obscured in one or several different time periods, resulting in the inability to capture the individual's facial expression in one or several frames.

[0097] Optionally, in this embodiment, the individual emotion calculation method based on time consistency can be implemented by defining a panic index.

[0098] The first is individual facial expression emotion recognition based on temporal consistency. If an individual a's facial expression is obscured in one or several different time periods, it will be impossible to collect the facial expression of the individual in one or several frames.

[0099] In order to solve this problem, we define an index P to determine whether an individual is in a panic mood. For the jth individual, its panic mood index P is j The calculation of is as follows:

[0100]

[0101] Here, m is the total number of frames that the video is divided into, the denominator represents the total number of frames in which individual j can collect complete facial expression information, that is, the number of frames in which the facial image of the person whose emotion can be accurately identified, and the numerator represents the number of frames in which individual j's facial image is identified as panic.

[0102] The final emotion of an individual is r. For individual j, the final emotion of individual j is r j , r j The expression is as follows:

[0103]

[0104] In this implementation, it is assumed that P≥50%, that is, r j = -1, it means that the number of frames that can be identified as panic emotions in all frames of complete facial expression information collected by an individual accounts for more than half of all frames of complete facial expression information collected by the individual. In this case, the individual can be defined as being in panic in the crowd movement video. If P < 50%, that is, r j =1, it means that the number of frames identified as panic emotions among all the frames in which complete facial expression information can be collected for an individual accounts for less than half of all the frames in which complete facial expression information can be collected for the individual. In this case, the individual can be defined as being in a non-panic emotion in the crowd motion video.

[0105] In some embodiments, a method for calculating whether the occluded individuals in all frame images are in a panic mood based on spatial consistency may be: converting each frame of the image into a depth map; calculating the distance between all individuals in each frame image for which emotional results have been calculated and the occluded individual j, obtaining the sum of the influence values ​​of the emotions of individual j on the emotions of other individuals in the group based on the distance, and determining whether the emotion is panic based on the influence value, where the influence of other individuals on the panic emotion of individual j is a negative value, and the influence of non-panic emotion is a positive value.

[0106] When an individual j is occluded in all frames, each frame is converted into a depth map to determine the distance between all individuals in the frame image whose emotional results have been calculated and individual j. Based on the distance between individuals, the panic impact of the surrounding individuals on the occluded individual j is obtained. The emotion of the occluded individual is the weighted sum of the panic impact of all other individuals on the occluded individual.

[0107] Individual j's emotions are influenced by other individuals in the group, and this influence is related to the distance between them. The influence of other individuals on individual j's panic is negative, while the influence of non-panic emotions is positive. Ultimately, the influence of surrounding individuals on individual j's emotions is the sum of the influences of each individual.

[0108] A depth image, also known as a range image, is an image that uses the distance (depth) from the image collector to each point in the scene as its pixel value. It directly reflects the geometric shape of the visible surface of the scene. Depth images can be calculated as point cloud data through coordinate transformation. Point cloud data with regularity and necessary information can also be inversely calculated into depth image data. A depth map is an image or image channel that contains information about the distance from the surface of scene objects to the viewpoint and is used to simulate 3D shapes or reconstruct them.

[0109] Depth images can be represented as pseudo-grayscale, point clouds, or meshes. Local feature extraction, a key step in 3D modeling and object recognition, involves both feature detection and feature description. In this example, the depth map can be used to determine the longitudinal distance between entities. Using the directly derived lateral distance, the distance between any two entities can be calculated using the Pythagorean theorem.

[0110] Depth maps are captured by a camera, and each pixel represents the distance between the object and the camera in the xy plane. The unit is mm. Through machine learning, a large number of training images and corresponding depth images are fed into a defined model for supervised learning. Once the model is trained, the actual test image can be fed into the trained model for depth calculation.

[0111] In the image frame provided by the depth data stream, each pixel represents the distance (in millimeters) from the object at that specific (x, y, z) coordinate in the depth sensor's field of view to the nearest object in the camera space.

[0112] In the depth map, the distance d between individual s and individual j sj ,as follows:

[0113]

[0114] Among them, j is the individual whose facial expression is completely blocked during the entire time period, and the position coordinates of j are (x j ,y j ,z j ). s is any individual in the entire group whose emotions can be recognized by facial expressions. Define s as the neighbor of j, and the position coordinates of s are (x s ,y s ,z s ).

[0115] Among them, r s represents the final emotion of individual s, j is the individual whose facial expression is completely blocked during the entire time period, and s is the individual in the entire group whose emotion can be identified through facial expression. sj is the weight of the influence of individual s on individual j’s emotions, and this weight is related to the distance d between the two individuals. sj The greater the distance between two individuals, the smaller the weight of this influence; the smaller the distance between two individuals, the greater the weight of this influence. sj is represented as follows:

[0116]

[0117] If an individual's facial expression is completely obscured during the entire time period, then the individual's emotion is affected by all individuals in the group whose emotions have been identified. Define an index V to determine whether the emotion of an individual whose facial expression is completely obscured during the entire time period is panic. For individual j, its panic index V is calculated as follows:

[0118]

[0119] Among them, individual s is the individual whose emotion can be directly identified through facial expression and the final emotion individual is calculated through time consistency. s =1, indicating that the emotion of individual s is non-panic, and the value of the influence of individual s on the non-panic emotion of individual j is positive; rs = -1, indicating that the emotion of individual s is panic, and the value of the influence of individual s on the panic emotion of individual j is negative.

[0120] In this embodiment, it is defined that when V≤0, the emotion of individual j is determined to be panic; when V>0, the emotion of individual j is determined to be non-panic.

[0121] The calculation of individual emotions based on temporal consistency and spatial consistency is performed on formula (4)-formula (14) and matrix R. This method can accurately obtain the final emotion of an individual when facial information is incomplete or even cannot be collected due to occlusion.

[0122] In this embodiment, individual facial expression emotion recognition based on spatial consistency solves the problem that an individual's facial expression may be blocked for an entire period of time, resulting in the inability to capture the individual's facial expression in any frame.

[0123] In step 3, the concept of group panic degree is defined to calculate the group facial expression emotion. After obtaining the final emotion of the individual, the group panic degree is defined based on the percentage of the number of individual panic emotions in the group to describe the index of the panic emotion of the group, and finally the true emotion of the group is obtained. The specific group panic degree is as follows:

[0124] For the group to be detected, the group panic level includes no panic, mild panic, moderate panic and severe panic, which is determined by describing the crowd panic level index S, where the crowd panic level index S is:

[0125]

[0126] Among them, n represents the total number of people in the video, n p Indicates the number of people whose facial expressions in the video were identified as panic.

[0127] Table 1 shows the definition and judgment criteria of the panic degree of the crowd in this embodiment. No panic means that no individual in the crowd to be detected has panic emotions, that is, S=0; mild panic means that the number of people in the crowd to be detected who have panic emotions accounts for 0-30% of the total number of the crowd, that is, S∈(0,0.30]); moderate panic means that the number of people in the crowd to be detected who have panic emotions accounts for 30%-60% of the total number of the crowd, that is, S∈(0.30,0.60]; severe panic means that the number of people in the crowd to be detected who have panic emotions accounts for 60%-100% of the total number of the crowd, that is, S∈(0.60,1].

[0128] Table 1

[0129] Panic No panic Mild panic Moderate panic Severe panic expression S＝0 S∈(0,0.30] S∈(0.30,0.60] S∈(0.60,1]

[0130] The above method can be used to determine the panic level of the overall mood of a group of people. An evacuation plan can be deployed based on the panic level to ensure the safety of the group to the greatest extent possible.

[0131] Example 2

[0132] Based on Example 1, this embodiment provides a group facial expression and emotion recognition system based on spatiotemporal consistency, including:

[0133] Face recognition module: configured to perform face detection and recognition on the acquired video images of the crowd motion scene to be recognized, and obtain the facial expression of each individual in each frame of the image;

[0134] Individual panic judgment module: This module is configured to identify whether the individual's emotion is panic based on the facial expression of each individual; calculate whether the individual occluded in some frames is panic based on temporal consistency, and calculate whether the individual occluded in all frames is panic based on spatial consistency;

[0135] The group panic degree judgment module is configured to obtain the group panic degree according to the proportion of panicking individuals among all individuals.

[0136] Example 3

[0137] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the method of embodiment 1 are completed.

[0138] Example 4

[0139] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps described in the method of embodiment 1 are completed.

[0140] The electronic devices proposed in the present disclosure can be mobile terminals and non-mobile terminals. Non-mobile terminals include desktop computers, and mobile terminals include smart phones (such as Android phones, IOS phones, etc.), smart glasses, smart watches, smart bracelets, tablets, laptops, personal digital assistants, and other mobile Internet devices that can communicate wirelessly.

[0141] It should be understood that in the present disclosure, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0142] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0143] During implementation, each step of the above method can be performed by hardware integrated logic circuits in a processor or by software instructions. The steps of the method disclosed in this disclosure can be directly implemented as execution by a hardware processor, or by a combination of hardware and software modules in the processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in a memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here. Those skilled in the art will appreciate that the units, i.e., algorithmic steps, of the various examples described in connection with the embodiments disclosed herein can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this disclosure.

[0144] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0145] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a division of logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0146] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0147] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

[0148] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A group expression emotion recognition method based on spatiotemporal consistency, characterized by: The steps include: Perform face detection and recognition on the acquired video images of the crowd motion scene to be identified to obtain the facial expression of each individual in each frame of the image; Identify whether the individual's emotion is panic based on the facial expression of each individual; Based on temporal consistency, we calculate whether the individuals occluded in some frames are in panic. Based on spatial consistency, we calculate whether the individuals occluded in all frames are in panic. The panic degree of the group is obtained according to the proportion of panic individuals among all individuals; A method for calculating whether an occluded individual a in a partially obscured frame image is panicking based on temporal consistency is as follows: if the proportion of frames in which the complete facial expression information of individual a is captured exceeds a set threshold, the individual a is considered panicking in the video of the identified crowd movement; otherwise, the individual is considered non-panic. Based on spatial consistency, the method calculates whether the occluded individual j in all frames is in a panic state. Specifically, each frame is converted into a depth map. The distance between all individuals whose emotions have been calculated in each frame and the occluded individual j is calculated. Based on the distance, the sum of the influence values ​​of individual j's emotions on other individuals in the group is obtained. Based on the influence value, whether the individual j is in a panic state is determined. The influence of other individuals on individual j's panic state is negative, and the influence of non-panic state is positive. In the depth map, the distance d between individual s and individual j sj ,as follows: Among them, j is the individual whose facial expression is completely blocked during the entire time period, and the position coordinates of j are (x j ,y j ,z j ); s is any individual in the entire group whose emotions can be recognized by facial expressions; define s as a neighbor of j, and the position coordinates of s are (x s ,y s ,z s ); Among them, r s represents the final emotion of individual s, j is the individual whose facial expression is completely blocked during the entire time period, and s is the individual in the entire group whose emotion can be recognized through facial expression; w sj is the weight of the influence of individual s on individual j’s emotions; w sj is represented as follows: Define an index V for whether an individual whose facial expression is completely obscured during the entire time period is considered to be in a panic mood. For individual j, the panic mood index V is calculated as follows: Among them, individual s is the individual whose emotion can be directly recognized by facial expression and the final emotion individual is calculated by time consistency; r s =1, indicating that the emotion of individual s is non-panic, and the value of the influence of individual s on the non-panic emotion of individual j is positive; rs = -1, indicating that the emotion of individual s is panic, and the value of the influence of individual s on the panic emotion of individual j is negative; When V≤0, the emotion of individual j is judged as panic; when V>0, the emotion of individual j is judged as non-panic; The degree of group panic includes no panic, mild panic, moderate panic and severe panic.

2. The method for group facial expression recognition based on spatiotemporal consistency according to claim 1, wherein: A method for performing face detection and recognition on a video image of a crowd motion scene to be recognized to obtain the facial expression of each individual in each frame of the image comprises the following steps: Input a video of a crowd motion scene to be identified; Divide the video into frames of pictures; Perform face detection on each frame of the image processed by frame; Crop the faces detected in each picture to get a picture containing only the face; The cropped face image is input into the deep residual network to obtain the facial expression recognition result.

3. The method for group facial expression recognition based on spatiotemporal consistency according to claim 2, wherein: The face detection for each frame of the image is realized by using the Haar feature extraction algorithm; or, The steps to crop the detected face to obtain a picture containing only the face are as follows: For each picture after face detection, set a rectangular frame close to the size of the face to select the detected face; Crop out the faces selected in each picture and label them, and output each face in a picture in the order of the labels.

4. The method for group facial expression recognition based on spatiotemporal consistency according to claim 2, wherein: For the facial expression recognition results obtained in the deep residual network, the emotion recognition result matrix is ​​obtained in the form of a matrix; Alternatively, the deep residual network is a Resnet34 network; The Resnet34 network structure includes a convolutional layer, a residual module, a pooling layer, a fully connected layer and a Softmax classifier, and the convolutional layer, the residual module, the pooling layer, the fully connected layer and the Softmax classifier are connected in sequence.

5. A group expression and emotion recognition system based on spatiotemporal consistency, characterized by: include: Face recognition module: configured to perform face detection and recognition on the acquired video images of the crowd motion scene to be recognized, and obtain the facial expression of each individual in each frame of the image; Individual panic judgment module: configured to identify whether the individual's emotion is panic based on the facial expression of each individual; calculate whether the individual occluded in some frame images is panic based on temporal consistency, and calculate whether the individual occluded in all frame images is panic based on spatial consistency; A group panic degree judgment module is configured to obtain a group panic degree based on the proportion of panicking individuals among all individuals; A method for calculating whether an occluded individual a in a partially obscured frame image is panicking based on temporal consistency is as follows: if the proportion of frames in which the complete facial expression information of individual a is captured exceeds a set threshold, the individual a is considered panicking in the video of the identified crowd movement; otherwise, the individual is considered non-panic. Based on spatial consistency, the method calculates whether the occluded individual j in all frames is in a panic state. Specifically, each frame is converted into a depth map. The distance between all individuals whose emotions have been calculated in each frame and the occluded individual j is calculated. Based on the distance, the sum of the influence values ​​of individual j's emotions on other individuals in the group is obtained. Based on the influence value, whether the individual j is in a panic state is determined. The influence of other individuals on individual j's panic state is negative, and the influence of non-panic state is positive. In the depth map, the distance d between individual s and individual j sj ,as follows: Among them, j is the individual whose facial expression is completely blocked during the entire time period, and the position coordinates of j are (x j ,y j ,z j ); s is any individual in the entire group whose emotions can be recognized by facial expressions; define s as a neighbor of j, and the position coordinates of s are (x s ,y s ,z s ); Among them, r s represents the final emotion of individual s, j is the individual whose facial expression is completely blocked during the entire time period, and s is the individual in the entire group whose emotion can be recognized through facial expression; w sj is the weight of the influence of individual s on individual j’s emotions; w sj is represented as follows: Define an index V for whether an individual whose facial expression is completely obscured during the entire time period is considered to be in a panic mood. For individual j, the panic mood index V is calculated as follows: Among them, individual s is the individual whose emotion can be directly recognized by facial expression and the final emotion individual is calculated by time consistency; r s =1, indicating that the emotion of individual s is non-panic, and the value of the influence of individual s on the non-panic emotion of individual j is positive; rs = -1, indicating that the emotion of individual s is panic, and the value of the influence of individual s on the panic emotion of individual j is negative; When V≤0, the emotion of individual j is judged as panic; when V>0, the emotion of individual j is judged as non-panic; The degree of group panic includes no panic, mild panic, moderate panic and severe panic.

6. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of any one of the methods of claims 1 to 4 are completed.

7. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of any one of the methods of claims 1 to 4.

Images