Emotion recognition method, device, equipment, medium and product

Abstract

The invention discloses an emotion recognition method, device and equipment, a medium and a product. The method comprises the following steps: acquiring a to-be-processed video comprising at least one to-be-detected object in a to-be-detected area and object feature data of the to-be-detected object; the video to be processed comprises a plurality of video frames; performing space-time normalization processing on the plurality of video frames to obtain a face image and a trunk image corresponding to each video frame; inputting the face image into a trained face processing model to obtain a face feature map, and inputting the trunk image into a trained trunk processing model to obtain a trunk feature map; and for each to-be-detected object, processing the object feature data of the to-be-detected object to obtain an object emotional feature, and determining emotional data corresponding to the to-be-detected object based on the object emotional feature, the facial feature map and the trunk feature map. According to the technical scheme provided by the invention, the effect of improving the emotion recognition accuracy is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer processing, and in particular to an emotion recognition method, device, equipment, medium and product. BACKGROUND

[0002] With the improvement of smart tourism and public safety management, emotion recognition technology plays an increasingly important role in the response to emergencies in scenic spots. By detecting the emotional state of tourists (such as anxiety, panic, anger) in real time, the management system can issue timely warnings, optimize evacuation plans or start psychological comfort measures, which helps to reduce chaos and ensure safety.

[0003] In the prior art, the method of emotion recognition mainly relies on machine learning method to process facial images, extract facial features of the object and combine facial key points for emotion classification. This kind of method is easily disturbed by the background where the object is located, and has the problem of low emotion recognition accuracy. SUMMARY

[0004] The present application provides an emotion recognition method, device, equipment, medium and product to realize the technical effect of improving the accuracy of emotion recognition.

[0005] According to an aspect of the present application, an emotion recognition method is provided, which comprises:

[0006] Obtaining a to-be-processed video including at least one to-be-detected object in a to-be-detected area and object feature data of the to-be-detected object; the to-be-processed video includes a plurality of video frames;

[0007] Performing spatio-temporal normalization processing on the plurality of video frames to obtain a facial image and a torso image corresponding to each video frame;

[0008] Inputting the facial image into a trained facial processing model to obtain a facial feature map, and inputting the torso image into a trained torso processing model to obtain a torso feature map;

[0009] For each to-be-detected object, processing the object feature data of the to-be-detected object to obtain an object emotion feature, and determining emotion data corresponding to the to-be-detected object based on the object emotion feature, the facial feature map and the torso feature map.

[0010] According to another aspect of the present application, an emotion recognition device is provided, which comprises:

[0011] A data acquisition module is configured to acquire a to-be-processed video including at least one to-be-detected object in a to-be-detected area and object feature data of the to-be-detected object; the to-be-processed video includes a plurality of video frames;

[0012] The normalization processing module is configured to perform spatio-temporal normalization processing on the plurality of video frames to obtain a face image and a torso image corresponding to each of the video frames.

[0013] The feature map determination module is configured to input the face image into a trained face processing model to obtain a face feature map, and input the torso image into a trained torso processing model to obtain a torso feature map.

[0014] The emotion data determination module is configured to, for each of the to-be-detected objects, process object feature data of the to-be-detected object to obtain object emotion features, and determine emotion data corresponding to the to-be-detected object based on the object emotion features, the face feature map, and the torso feature map.

[0015] According to another aspect of the present application, an electronic device is provided, which comprises:

[0016] at least one processor; and a memory connected to the at least one processor in communication; wherein

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the emotion recognition method according to any one of the embodiments of the present application.

[0018] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to perform the emotion recognition method according to any one of the embodiments of the present application when executed by the processor.

[0019] According to another aspect of the present application, a computer program product is provided, which comprises a computer program for enabling a processor to perform the emotion recognition method according to any one of the embodiments of the present application when executed by the processor.

[0020] The technical scheme of the embodiment of the present application comprises the following steps: obtaining a to-be-processed video including at least one to-be-detected object in a to-be-detected region and object feature data of the to-be-detected object; the to-be-processed video comprises a plurality of video frames; performing space-time normalization processing on the plurality of video frames to obtain a face image and a torso image corresponding to each video frame; inputting the face image into a trained face processing model to obtain a face feature map, and inputting the torso image into a trained torso processing model to obtain a torso feature map; for each to-be-detected object, processing the object feature data of the to-be-detected object to obtain object emotion features, and determining emotion data corresponding to the to-be-detected object based on the object emotion features, the face feature map and the torso feature map, thereby solving the problem of low recognition accuracy in the prior art which only relies on recognizing object face features for emotion recognition, realizing space-time normalization processing on a plurality of video frames in the to-be-processed video to extract standardized face images and torso images, inputting the face images and the torso images into the trained face processing model and the trained torso processing model respectively to obtain high-discriminative face feature maps and torso feature maps, processing the object feature data of the to-be-detected object to extract object emotion features, and then performing collaborative fusion analysis on the face and torso dynamic features in the visual mode and the non-visual object emotion features in the decision-making stage, realizing joint modeling of multiple dimensions and multiple sources from a single mode, enhancing the recognition ability of complex emotion states (such as suppressed anger and pseudo joy), improving the robustness in actual scenes such as low light, occlusion or long distance, and improving the accuracy and reliability of emotion recognition, so as to effectively calm the to-be-detected object based on the emotion data.

[0021] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is a flowchart of an emotion recognition method according to an embodiment of the present application;

[0024] Figure 2 is a flowchart of an emotion recognition method according to an embodiment of the present application;

[0025] Figure 3 is a flowchart of an emotion recognition method according to an embodiment of the present application;

[0026] Figure 4 is a structural schematic diagram of an emotion recognition device according to an embodiment of the present application;

[0027] Figure 5 is a structural schematic diagram of an electronic device for implementing an emotion recognition method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] It should be noted that in the technical scheme of the present disclosure, the collection, collection, updating, analysis, processing, use, transmission, storage and the like of user personal information are in line with the relevant legal regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken to prevent illegal access to user personal information data, and to maintain user personal information security and network security.

[0031] Figure 1 is a flowchart of an emotion recognition method according to an embodiment of the present application. The present embodiment can be applicable to the case of identifying the emotion of an object. The method can be executed by an emotion recognition device, which can be realized in the form of hardware and / or software, and can be configured in a computing device. As shown in the figure, the method comprises: Figure 1

[0032] ​S110, acquire a to-be-processed video including at least one to-be-detected object in a to-be-detected region and object feature data of the to-be-detected object; the to-be-processed video includes a plurality of video frames.

[0033] The to-be-detected region can refer to a region in which the emotion of the to-be-detected object needs to be recognized. The number of to-be-detected regions includes one or more. The to-be-detected object can refer to an individual whose emotion needs to be detected. For example, the to-be-detected object can be a person, an animal, or other targets. The to-be-processed video can be a video stream or a video clip obtained by capturing the to-be-detected region. The to-be-processed video can include not only the to-be-detected object but also environmental background information, building facilities, and the like. The to-be-processed video is composed of a series of continuous video frames, and each video frame can be an image. For example, the to-be-processed video can be played at a speed of several frames per second (for example, 25 fps represents 25 frames per second), and a plurality of continuous frames reflect the change process of the to-be-detected object over time. The object feature data can refer to other modal information related to the to-be-detected object in addition to the video.

[0034] In the embodiment, the camera can be pre-deployed in the to-be-detected region. In order to ensure that there is enough spatiotemporal information to analyze the dynamic behavior characteristics (such as expression changes and action processes) of the object in the to-be-detected region, the camera can capture the to-be-detected region to obtain the video, and read the captured video as the to-be-processed video in real time or periodically. The time length of the to-be-processed video can be 3 seconds, 5 seconds, 10 seconds, or the like, and the time length is not limited. The to-be-processed video can include one or more to-be-detected objects. The object feature data of the to-be-detected object is acquired. For example, the object feature data includes but is not limited to physiological signals, facial features, sound features, user portrait information, and the like. So that subsequent emotion states of the to-be-detected object can be detected based on the to-be-processed video and the object feature data, and the management system can timely issue a warning, optimize an evacuation plan, or start a psychological comfort measure.

[0035] For example, when the camera captures the to-be-detected region to obtain the video, when the to-be-detected object appears in the video (for example, a person enters the field of view by detecting an algorithm), the timing starts. All video frames are continuously recorded to form a video clip, and the video clip is taken as the to-be-processed video. The object feature data of the to-be-detected object can be synchronously collected by various sensors. For example, the sensors include but are not limited to wearable devices, audio collection devices, and the like. All object feature data can be time-synchronized with the video frames in the to-be-processed video to ensure that the data at the same time corresponds to determine the emotion recognition accuracy.

[0036] It should be noted that in the technical solutions of the present disclosure, the collection, collection, update, analysis, processing, use, transmission, storage, and the like of user personal information are performed with the knowledge and permission of the user, and relevant privacy protection regulations are followed.

[0037] S120, performing spatio-temporal normalization processing on the plurality of video frames to obtain a face image and a torso image corresponding to each video frame.

[0038] The face image can refer to a face region image cropped and standardized from each video frame. The torso image can refer to an upper body region image (e.g., including shoulders, chest, arms, etc.) extracted and standardized from each video frame.

[0039] In this embodiment, the optical flow or background registration technology can be used to process part or all of the video frames to eliminate the inter-frame displacement caused by camera shaking or slight shaking of the subject, and eliminate the differences caused by shooting distance, angle, motion, etc., so that the face and torso are stable in position in the sequence, and the real behavior change is highlighted. Further, a face detection algorithm (such as MTCNN, RetinaFace) can be used to locate the face region in the processed video frame. A human body detection or pose estimation algorithm can be used to locate the torso region in the processed video frame. The extracted face region and torso region can also be scaled to a fixed size (such as face 112x112, torso 224x224), and an interpolation method can be used to ensure clarity. Histogram equalization, white balance adjustment, etc. can also be used to adjust the illumination and color consistency of the scaled face region and torso region. The adjusted face region is used as the face image, and the adjusted torso region is used as the torso image, reducing the influence of illumination changes. Expression, pose or behavior analysis is performed based on the extracted face image and torso image.

[0040] In order to extract high-quality, standardized face and torso region images from the original video frames, in the process of performing spatio-temporal normalization processing on the plurality of video frames to obtain a face image and a torso image corresponding to each video frame: the plurality of video frames can be aligned to obtain a plurality of first images; for each first image, the pixel color data in the first image is converted to obtain a second image, and the second image is enhanced to obtain a third image; based on a region detection algorithm, the third image is detected to obtain a face image and a torso image in the third image.

[0041] The first image can refer to a video frame after alignment processing, which reduces motion interference compared to the original video frame. The second image can refer to an image after color conversion of the first image. For example, after converting the first image from RGB to Lab, a new image containing brightness (L) and two color channels (a, b) is obtained as the second image. The third image can refer to an image after enhancement of the second image. The region detection algorithm is used to identify specific target regions (such as face, upper body) in the image. For example, the region detection algorithm can be a deep learning or image analysis based algorithm model.

[0042] In the present embodiment, the optical flow method or feature point matching technology can be used to analyze the motion of pixels between adjacent video frames, and the multiple video frames are aligned to obtain multiple first images. For example, the translation, rotation and even affine deformation parameters of each frame relative to the reference frame (such as the first frame or the middle frame) can be calculated; the reverse transformation is applied to each frame to align it to the unified view angle and position, thereby eliminating the displacement, rotation or deformation between the video frames caused by the camera shaking, object moving or posture changing, so that the position and posture of the same target (such as the face or the limb) in different frames remain consistent, and a set of time-continuous and space-stable image sequences (i.e. first images) are output. Further, the pixel color data of each first image can be converted from the RGB color space to the Lab color space (or other color space) to obtain multiple second images, so as to separate the brightness (L) and color information (a, b) of the pixels in the first image, wherein the a channel reflects the change from green to red, and the b channel reflects the change from blue to yellow. Through the conversion processing, the slight color fluctuation of the face caused by the emotional change (such as flushing and pallor) in the image can be captured, and the interference of the light can be avoided. Further, the histogram equalization or contrast stretching can be performed on the a and b color channels of the second image to enhance the saliency of the skin color change, and the adaptive enhancement algorithm (such as CLAHE) can be combined to avoid over-amplification of noise, and the image after the enhancement processing is taken as a third image, so that the third image can more clearly show the expression-related physiological response (such as facial congestion when nervous), and the accuracy of the subsequent detection and recognition is improved. Based on the face detection algorithm in the region detection algorithm, the face region in the third image is located, and the face image is cropped. At the same time, the human body posture estimation model or the human body detector in the region detection algorithm can be used to locate the upper body key points (such as shoulders and chest) in the third image, determine the trunk region, and crop the trunk image based on the trunk region.

[0043] For example, Figure 2Take the detection area as a scenic spot as an example, a 4K high-definition camera array (frame rate 30 fps) deployed at key positions in the scenic spot can be used to obtain a video stream including the to-be-detected object (such as a tourist) as a to-be-processed video; the to-be-processed video is converted into a video frame sequence including a plurality of images. At the same time, object feature data (such as voice data and other information related to the to-be-detected object) of the to-be-detected object can be collected through a wearable device and a directional microphone array. These multi-modal data (including the to-be-processed video and the object feature data) are sent into a preprocessing pipeline for spatio-temporal normalization processing of the to-be-processed video, including three steps: in the time dimension, a modified optical flow estimation algorithm (based on the Farneback method) is used to align a plurality of continuous video frames in the to-be-processed video, eliminating motion blur caused by head shaking, and the optical flow calculation uses a pyramid level of 3 and a window size of 15x15 pixels; in the spatial dimension, an adaptive region detection algorithm is used to determine the face ROI (i.e. face image) and the upper body region (i.e. torso image) in the aligned plurality of video frames, and the face image and the torso image are respectively scaled to two scales of 112x112 and 224x224 through bilinear interpolation; in the color space conversion, after the RGB data in the face image and the torso image are converted to the Lab color space, the a and b channels are enhanced based on histogram equalization, with enhancement coefficients a = 1.2 and β = 0.8, to highlight subtle expression color changes. The voice data in the object feature data is extracted for voice features through MFCC feature extraction (26-dimensional coefficients), and other information related to the to-be-detected object can be extracted for time-frequency features after Butterworth band-pass filter (0.1-5 Hz) denoising; the voice features and the time-frequency features are used as object emotion features. All modal data (including object emotion features, face images and torso images) are finally unified to the same time reference (with a time synchronization error of less than 10 ms), so as to determine the emotion data of the to-be-detected object based on the standardized object emotion features, face images and torso images.

[0044] S130, input the face image into the trained face processing model to obtain a face feature map, and input the torso image into the trained torso processing model to obtain a torso feature map.

[0045] The face processing model can be used to extract features related to expressions and emotions from the face image, and output a face feature map. The face feature map can be a high-dimensional semantic feature representation extracted from the face image by the face processing model, and can be represented in the form of a multi-channel tensor. The face feature map can include visual features related to expressions, eye contact, mouth movements, etc. The torso processing model can be used to extract features related to limb movements from the torso image, and output a torso feature map. The torso feature map can be a structured feature representation extracted from the torso image (e.g., the upper body). The torso feature map can include features related to body language such as posture, gestures, limb movements, etc. The torso feature map can reflect the external manifestations of the subject's emotions on the body posture, such as crossed arms when nervous, natural drooping when relaxed, etc.

[0046] In this embodiment, the face image can be used as input data for the trained face processing model, and the face image can be passed through the face processing model layer by layer, and can be subjected to convolution, activation, pooling, etc. to extract high-level semantic features from the face image, such as eyebrow center wrinkling, mouth corner opening, skin color change, etc. features related to emotions and intentions, and output a face feature map. Similarly, the torso image can be used as input data for the trained torso processing model, and the model can analyze the object's torso structure and spatial relationship in the torso image, such as arm position, body tilt angle, movement trend, etc. features related to posture and movement, and output a torso feature map. It should be noted that a plurality of consecutive face images can be input into the face processing model to obtain a face feature map. A plurality of consecutive torso images can also be input into the torso processing model to obtain a torso feature map, in order to improve the accuracy of emotion recognition.

[0047] In order to more accurately capture the subtle expression changes of the face in the time series (such as smiling, frowning, etc. non-rigid motion), in this embodiment, the face processing model includes at least a first convolutional layer, an attention module, a fully connected layer, and a second convolutional layer. Accordingly, the face image is input into the trained face processing model to obtain a face feature map, including: sampling the face image based on the first convolutional layer to obtain local picture features; processing the local picture features based on the attention module to obtain channel descriptors; processing the channel descriptors based on the fully connected layer to obtain channel weights; and processing the face image based on the channel weights to obtain the face feature map.

[0048] The first convolutional layer can be a convolutional operation layer, which is used to extract basic local visual features such as edges, corner points, textures, etc. from the input face image. The local picture feature refers to the feature output by the first convolutional layer, which represents each sub-region (such as the eye corner, nose wing, and mouth corner) in the face image. The local picture feature can reflect the local structure information in the face image. The attention module can be used to analyze the local picture feature and determine the importance of each feature channel. Each feature channel is used to independently learn a response type feature such as color change, edge, texture, or contour, etc. The channel descriptor is a statistical summary of the local picture feature in the spatial and temporal dimensions, which is used to represent the overall activity of the corresponding feature channel. The fully connected layer is used to non-linearly combine and map the features of the previous layer, and predict the importance of each feature channel according to the channel descriptor. The channel weight can be a numerical vector, which is used to represent the importance attribute value of each feature channel. The feature channel with a higher weight is considered to contribute more to the current feature processing task (such as expression recognition).

[0049] In this embodiment, the face image can be input to the first convolutional layer, which performs local sampling on the face image to obtain local picture features that retain spatial structure information. Further, the local picture features are input into the attention module, which is compressed in the spatial dimension (height and width) and the temporal dimension. The sum of all picture pixels (i.e. feature values) in the same feature channel of the local picture feature is obtained, and the product value between the spatial dimension and the temporal dimension is determined. The quotient value between the sum value and the product value is taken as the channel descriptor of the feature channel. Accordingly, the channel descriptor of each feature channel can be obtained. The channel descriptor is input into the fully connected layer, which can use a non-linear activation function (such as ReLU) to transform the channel descriptor. The output result is limited to between 0 and 1 by a Sigmoid function, and the output result is the channel weight of the feature channel. The channel weights of different feature channels can be multiplied with the feature values of the corresponding feature channels in the original local picture feature channel by channel. The feature channels with high weights are enhanced, and the feature channels with low weights are suppressed, achieving automatic focusing. After this multiplication operation, a face feature map is obtained. This method can automatically focus on the key regions of the face (such as the eyes and mouth), thereby generating a more discriminative face feature map.

[0050] For example, the facial processing model is a branch of facial processing. The facial processing model can receive a 112×112×16×3 facial image (the first 112 refers to the height; the second 112 refers to the width; 16 refers to the number of video frames in the time dimension; 3 indicates three Lab channels). After passing through a 3D convolutional layer (i.e., the first convolutional layer, kernel size 5×5×3, stride 1×1×1, padding 2×2×1), preliminary feature extraction is performed on the facial image, outputting a 64-channel feature map (i.e., local image features). The convolution formula for this first convolutional layer can be expressed as: ;in, It is the local image feature output after convolution calculation at the location Eigenvalues ​​at; It refers to the current position coordinates of the currently calculated feature value on the local image feature in the output; This refers to the number of sampling points contained within the convolution kernel, which determines the number of position points that need to be considered during the convolution process; It refers to the weight value corresponding to the nth sampling point in the convolution kernel. The weight is used to perform weighted summation of the input features in the convolution operation. Represents a facial image; It refers to the original position coordinates of the nth sampling point within the convolution kernel, which defines the position of the sampling point without offset; It is a key parameter introduced in deformable convolution, namely the two-dimensional offset of the nth sampling point, which can be predicted by additional convolutional layers. In other words, this technical solution adds a two-dimensional offset to the position of each sampling point. The kernel enumerates N locations. This design allows the convolutional network to adaptively adjust its receptive field, thereby better capturing the non-rigid deformations of facial muscle movements.

[0051] Next, the local image features can be input into the dual-path attention module. The attention path first compresses the local image features F∈R(C×T×H×W) along the spatiotemporal dimension to generate channel descriptors. The channel descriptors can be determined based on formula (1). Formula (1) can be expressed as: ;in, It can be represented as the first The channel descriptor of each feature channel; T refers to the time dimension of the local image feature; H refers to the height dimension of the local image feature; W refers to the width dimension of the local image feature; THW is the product of T, H, and W, and serves as the normalization coefficient after summing the spatiotemporal dimension feature values. Represented as local image feature F in the first... the feature value at the position of the t-th time dimension, the i-th height dimension, and the j-th width dimension of the t-th channel. Then, the channel descriptor can be processed by a two-level fully connected layer (intermediate layer dimension reduction ratio r = 16) to generate a channel weight; for example, the channel weight can be generated based on formula (2); formula (2) can be expressed as: ; wherein, can be expressed as the channel weight of the i-th feature channel; represents the weight learned by the fully connected layer; is a ReLU activation function, is a Sigmoid function.

[0052] In order to improve the representation ability of the key region of the face image, in the process of processing the face image based on the channel weight to obtain the face feature map: the face image can be weighted according to the channel weight to obtain a weighted picture feature; the weighted picture feature is processed by maximum pooling to obtain a first feature map, and the weighted picture feature is processed by average pooling to obtain a second feature map; the first feature map and the second feature map are processed based on the second convolutional layer to obtain a spatial weight; the weighted picture feature is weighted according to the spatial weight to obtain the face feature map.

[0053] The weighted picture feature can be the result obtained by applying the channel weight to the original face image, that is, the different feature channels in the face image are reweighted, and this way can preserve the spatial structure, enhance important features, and suppress irrelevant features. The first feature map and the second feature map are respectively the output results of the maximum pooling and the average pooling of the weighted picture feature. The spatial weight can be a spatial mask with the same size as the input image, and each pixel point corresponds to a weight value between 0 and 1, indicating the importance of the position. For example, the weight of the eye and mouth region is higher.

[0054] ​In the embodiment, the channel weight can be multiplied by the face image to obtain a weighted picture feature. The weighted picture feature is subjected to max-pooling processing to extract the maximum response value of each spatial position in all channels in the weighted picture feature, highlight the most significant local activation region (such as the mouth corner movement), and obtain a first feature map. The weighted picture feature is subjected to average-pooling processing to extract the average response value of each spatial position in all channels, obtain a second feature map, and enable the second feature map to reflect the overall distribution trend (such as the whole face blush) and retain smoother global information. Further, the first feature map and the second feature map can be spliced in the channel dimension to form a new multi-channel feature map. The multi-channel feature map is sent to a second convolutional layer, the second convolutional layer learns how to combine the most significant response and the average response to judge the importance of each spatial position, and the output is normalized by a Sigmoid function to output a two-dimensional (or multi-channel) spatial mask with a range of 0-1 as a spatial weight. The spatial weight is multiplied with the weighted picture feature element by element (for example, pixel by pixel). The region with high weight (such as the smiling mouth) is further enhanced, and the region with low weight (such as the background or static cheek) is suppressed. Finally, the face feature map is output, so that the face feature map has a dual attention mechanism of channel importance and spatial importance, has a stronger semantic expression ability, and improves the accuracy of emotion recognition.

[0055] For example, the picture feature value of the corresponding feature channel in the face image can be weighted based on the feature weight of each feature channel. The weighted picture feature F1 is obtained after the weighting processing; then, the spatial attention path is used to perform max-pooling and average-pooling on the weighted picture feature along the channel axis to obtain two spatio-temporal feature maps Smax, Savg∈R(T×H×W), which are a first feature map and a second feature map, respectively. The spatial weight Ws can be generated by 3D convolution (i.e., the second convolutional layer, the kernel size is 3×3×3) after the first feature map and the second feature map are fused. The weighted picture feature is weighted according to the spatial weight to obtain the final face feature map. For example, the face feature map can be determined based on formula (3); wherein formula (3) can be expressed as ; represents element-wise multiplication; F1 represents the weighted picture feature; and Ws represents the spatial weight.

[0056] It should be noted that the trunk processing model is a limb action branch, and the structure architecture of the trunk processing model is similar to that of the face processing model. The input data of the trunk processing model is a full-body posture sequence of 224×224×16×3 (i.e., multiple trunk images). The trunk processing model can use a larger convolution kernel (7×7×3) in the time dimension to capture a larger range of limb movement patterns. The way to obtain the trunk feature map is similar to the way to obtain the face image, which will not be described here.

[0057] In S140, for each to-be-tested object, the object feature data of the to-be-tested object is processed to obtain object emotion features, and emotion data corresponding to the to-be-tested object is determined based on the object emotion features, the face feature map and the torso feature map.

[0058] The object emotion features can be emotion-related features extracted by processing and analyzing the object feature data. The emotion data refers to data results about the emotional state of the to-be-tested object, and is used to describe the emotional state of the to-be-tested object. For example, the emotion data includes specific emotion types (such as happiness, sadness, anger, etc.), emotion intensity information, etc.

[0059] In this embodiment, a feature extraction algorithm or model can be used to analyze the object feature data, identify and quantify various emotion-related features of the object, and obtain the object emotion features. For example, an audio processing technology is used to analyze the voice information in the object feature data and identify the emotion information contained therein. The object emotion features include but are not limited to changes in facial expressions (such as smiling, frowning), changes in voice intonation, voice emotion features, changes in body posture, excitement level, heart rate changes, historical behavior patterns, etc. Further, the object emotion features, the face feature map and the torso feature map, etc. can be fused to obtain emotion data corresponding to the to-be-tested object, so that not only facial expressions but also body movements and object features, etc. are considered in multiple dimensions to determine the emotion data, and the understanding of emotions is more comprehensive and accurate.

[0060] For example, continuing to refer to Figure 2 The face feature map and the torso feature map are obtained by using a double-branch face processing model and a torso processing model to extract features from the face image and the torso image. The multi-modal features are obtained by fusing the object emotion features, the face feature map and the torso feature map, etc. The emotion data is output by inputting the multi-modal features into the emotion recognition module, and the emotion processing mode is determined based on the emotion data and the object density heat map.

[0061] In order to improve the robustness and accuracy of emotion recognition, in the process of determining the emotion data corresponding to the to-be-tested object based on the object emotion features, the face feature map and the torso feature map, the cross-attention processing is performed on the face feature map and the torso feature map to obtain to-be-fused feature data, the multi-modal feature data is determined based on the to-be-fused feature data and the object emotion features, and the emotion data corresponding to the to-be-tested object is determined based on the multi-modal feature data.

[0062] The multi-modal feature data refers to a unified feature representation obtained by fusing the visual modalities (face and torso) and the non-visual modalities (object emotion features).

[0063] In this embodiment, the face feature map and the torso feature map are taken as two independent input modalities, and a cross attention mechanism is constructed, e.g., taking the face feature map as "query", and taking the torso feature map as "key" and "value", to calculate to what extent the face region should pay attention to which spatial position of the torso; and vice versa. Through weighted summation of attention weights, two enhanced feature maps are generated: one is the face feature guided by the torso; and the other is the torso feature guided by the face. The two enhanced feature maps are spliced or added to form the to-be-fused feature data, which already contains cross-modal context interaction information and can capture the complex pattern of coordinated expression of facial expressions and body actions, realizing information complementation and enhancement. Further, the to-be-fused feature data (visual modality) is flattened or pooled into a one-dimensional vector to represent the overall visual emotional representation. The object emotional features (such as physiological features, speech features, etc.) are normalized and encoded to form another group of non-visual feature vectors. The two types of features are fused by feature splicing, gated fusion or attention weighting, etc., e.g., using a neural network to dynamically allocate weights of visual and non-visual features, to adaptively determine whether to believe eye contact or heart rate according to the current scene, and to output multi-modal feature data, which is a high-discriminative joint feature space representation. The multi-modal feature data is sent to a fully connected layer for classification after global spatio-temporal average pooling. If it is a classification task (such as identifying "angry", "happy", "sad"), the probability distribution of each emotional type can be output. The probability distribution includes the identification accuracy attribute of each emotional type, and the emotional type corresponding to the maximum identification accuracy attribute can be taken as the final emotional label. The final emotional label and the identification accuracy attribute are taken as the emotional data corresponding to the to-be-measured object. The entire probability distribution can also be taken as the emotional data, so that subsequent determination of how to calm the to-be-measured object can be combined with the overall emotional data. Time series modeling (such as LSTM) can also be combined to smooth the multi-frame corresponding emotional data, to improve the stability and continuity of emotion recognition.

[0064] The advantage of such a setting is that, through the cross attention mechanism, the face and torso features are mutually enhanced to capture the to-be-fused feature data of emotional expression, and then the to-be-fused feature data and the object emotional features are fused into multi-modal feature data for emotion recognition, to improve the robustness and accuracy of recognition.

[0065] In this embodiment, the face feature map and the torso feature map are cross-attention processed to obtain the to-be-fused feature data, including: based on a normalized exponential function, processing the face feature map and the torso feature map to obtain a cross-attention matrix; and based on the torso feature map, the face feature map and the cross-attention matrix, determining the to-be-fused feature data.

[0066] wherein the normalized exponential function (Softmax function) can be a mathematical function used to convert a set of real numbers into a probability distribution, whose output values are between 0 and 1 and the sum is 1. In the attention mechanism, the Softmax function is used to normalize the attention scores, so that they represent the relative importance weights between different positions. The cross-attention matrix is used to describe the two-dimensional weight matrix of the cross-modal association strength between the facial features and the torso features. Each element in the cross-attention matrix represents the attention degree of a certain region of the face to a certain region of the torso (or vice versa), reflecting the dynamic dependence relationship between the two modalities.

[0067] Specifically, the facial feature map and the torso feature map can be respectively mapped to the Query, Key, Value vector space through linear transformation. The facial feature map is used to generate Query, and the torso feature map is used to generate Key and Value (or vice versa, forming a bidirectional interaction). Next, the dot product similarity between Query and Key can be calculated to obtain an attention score matrix, which has a size of Hf x Wf x Hb x Wb (which can be flattened to N x M, where N = Hf x Wf is the number of face positions, and M = Hb x Wb is the number of torso positions). The attention score matrix reflects the correlation strength between each spatial position of the face and each position of the torso. The attention scores can be normalized by Softmax in the torso position dimension (i.e., M dimension), so that the attention weight corresponding to each face position sums to 1 over all torso positions, thereby obtaining a cross-attention matrix. Each element of the cross-attention matrix represents the relative importance weight of each torso region when the model focuses on a certain face region. Finally, the cross-attention matrix can be used to weight the feature values in the weighted torso feature map to generate a cross-modal feature that aligns with the spatial structure of the face feature map. The cross-modal feature and the face feature map are spliced or fused to obtain the to-be-fused feature data. The to-be-fused feature data fuses the torso context information most relevant to the current facial state. For example, when the face has a frown feature, the model can automatically enhance the attention on the torso regions such as forward leaning or crossed arms through the attention mechanism, thereby strengthening the judgment of the angry emotion. This enables the model to dynamically focus on the region in another modality (such as the torso) that is most relevant to its emotional expression, thereby achieving information complementation and semantic enhancement, and improving the robustness and accuracy of emotion recognition.

[0068] For example, the facial feature map F att and the torso feature map F btt of the body branch can be processed through a 1 x 1 x 1 convolutional layer to unify the number of channels. Then, the processed facial feature map F and the processed torso feature map F The cross-attention processing is performed to obtain a cross-attention matrix. The cross-attention matrix can be determined based on formula (4); formula (4) can be expressed as: ; wherein, represents the cross-attention matrix; is expressed as a normalized exponential function; ; ; and is a learnable parameter matrix, and d is a scaling factor. Further, based on the learnable parameter matrix , the torso feature map is weighted to obtain a first weighted feature, that is, the first weighted feature ; the first weighted feature is multiplied based on the cross-attention matrix to obtain a cross-modal feature. The cross-modal feature . The to-be-fused feature data is spliced with the face feature map to obtain the to-be-fused feature data. The to-be-fused feature data , is expressed as a splicing function.

[0069] The technical scheme provided in the embodiment is to obtain a to-be-processed video including at least one to-be-detected object in a to-be-detected region and object feature data of the to-be-detected object; the to-be-processed video includes a plurality of video frames; the plurality of video frames are subjected to spatiotemporal normalization processing to obtain a face image and a torso image corresponding to each video frame; the face image is input into a trained face processing model to obtain a face feature map, and the torso image is input into a trained torso processing model to obtain a torso feature map; for each to-be-detected object, the object feature data of the to-be-detected object is processed to obtain an object emotion feature, and based on the object emotion feature, the face feature map, and the torso feature map, emotion data corresponding to the to-be-detected object is determined, thereby solving the problem of low recognition accuracy in the prior art that only relies on recognizing object face features for emotion recognition, achieving spatiotemporal normalization processing of a plurality of video frames in a to-be-processed video to extract standardized face images and torso images, inputting the standardized face images and torso images into trained face processing models and torso processing models respectively to obtain high-discriminative face feature maps and torso feature maps, processing the object feature data of the to-be-detected object to extract an object emotion feature, and then performing collaborative fusion analysis of the face and torso dynamic features in the visual modality and the non-visual object emotion feature in the decision-making stage, thereby realizing joint modeling of multiple dimensions and multiple sources from a single modality, not only enhancing the recognition ability of complex emotional states (such as suppressed anger and pseudo joy), but also improving the robustness in actual scenes such as low light, occlusion, or long distance, and improving the accuracy and reliability of emotion recognition, so as to effectively calm the to-be-detected object based on the emotion data.

[0070] Figure 3is a flowchart of an emotion recognition method provided according to an embodiment of the present application. On the basis of the foregoing embodiment, the emotion data comprises recognition accuracy attributes corresponding to different emotion types; accordingly, an object density heat map can also be generated based on the torso image and / or the torso feature map; the emotion recognition confidence is determined based on the recognition accuracy attributes corresponding to different emotion types in the emotion data corresponding to the to-be-tested object; the emotion processing mode is determined based on the object density heat map, the emotion type and the emotion recognition confidence, so as to adjust the emotion of the to-be-tested object based on the emotion processing mode. The specific implementation can be referred to the technical solution of the present embodiment. The same or corresponding technical terms as those in the foregoing embodiment are not described herein.

[0071] As shown in Figure 3 , the method specifically comprises the following steps:

[0072] S210, generating an object density heat map based on the torso image and / or the torso feature map.

[0073] In the object density heat map, each pixel value can be used to represent the density of the object at the pixel position in the image space. For example, the higher the pixel value, the higher the density of the object at the pixel position in the space.

[0074] In the present embodiment, for the torso feature map corresponding to the same to-be-tested object, the global average pooling or the maximum pooling can be used to compress each frame of torso feature map in the channel dimension respectively, so as to retain the spatial response distribution and obtain a plurality of spatial response maps. The spatial response maps are subjected to pixel-by-pixel time accumulation or weighted average, so as to form a cross-frame spatio-temporal activation density map. The spatio-temporal activation density map can be subjected to Gaussian smoothing or normalization processing, so as to present a clear heat distribution form and obtain the object density heat map. Alternatively, the key point features in the torso image extracted by the OpenPose algorithm (a multi-person 2D human pose estimation algorithm based on a convolutional neural network) are used to generate the object density heat map based on the key point features. It should be noted that the OpenPose algorithm can be built into the torso processing model to participate in training together, so as to improve the accuracy of the heat map generation.

[0075] S220, determining the emotion recognition confidence based on the recognition accuracy attributes corresponding to different emotion types in the emotion data corresponding to the to-be-tested object.

[0076] The emotion data can be a probability distribution of a plurality of emotion types. The emotion types can include various types, for example, the emotion types can be anger, happiness, sadness, fear, surprise, normal, and the like. The recognition accuracy attribute can be used to represent the reliability of whether the emotion type actually exists. For example, the recognition accuracy attribute can be a classification probability value, the higher the recognition accuracy attribute, the more accurate the recognition, and vice versa, the lower the recognition accuracy attribute, the less accurate the recognition. The emotion recognition confidence can be used to represent the overall trust or accuracy of the emotion data recognition, or the emotion recognition confidence can also be used to represent the accuracy of the emotion type recognition with the maximum recognition accuracy attribute in the emotion data. For example, the higher the emotion recognition confidence, the more confident the emotion data, and the lower the emotion recognition confidence, the more likely there is a risk of misrecognition of the emotion data.

[0077] In this embodiment, the recognition accuracy attributes of different emotion types in the emotion data corresponding to the to-be-tested object can be processed based on the pre-trained machine learning model, and the emotion recognition confidence is outputted, so as to determine whether to perform emotion soothing on the to-be-tested object based on the emotion type, the emotion recognition confidence, and the object density heat map.

[0078] In this embodiment, the emotion recognition confidence is determined based on the recognition accuracy attributes of different emotion types in the emotion data corresponding to the to-be-tested object, including: determining a recognition uncertainty attribute based on the recognition accuracy attributes of different emotion types in the emotion data corresponding to the to-be-tested object; and determining the emotion recognition confidence based on the maximum recognition accuracy attribute and the recognition uncertainty attribute.

[0079] The recognition uncertainty attribute can be used to represent the overall uncertainty of the emotion data recognition.

[0080] In practical applications, the manner of determining the recognition uncertainty attribute comprises at least one of the following: measuring the recognition uncertainty attribute based on a difference between the maximum recognition accuracy attribute and the second largest recognition accuracy attribute, for example, the smaller the difference, the higher the uncertainty. For example, if, in addition to the maximum recognition accuracy attribute, other emotional attribute values are also high (such as the second highest close to the highest), it means that the system is uncertain about which emotional type it is, and the recognition uncertainty attribute is high; if the maximum recognition accuracy attribute is much higher than the recognition accuracy attributes of all other emotional types, it means that the system is clear in judgment, and the recognition uncertainty attribute is low. Alternatively, the Shannon entropy of the distribution formed by all recognition accuracy attributes is calculated, the larger the Shannon entropy, the more uniform the probability distribution, the higher the recognition uncertainty attribute, and the smaller the Shannon entropy, the smaller the recognition uncertainty attribute. Alternatively, after normalizing all recognition accuracy attributes into a probability distribution, the concentration degree thereof is analyzed to obtain the recognition uncertainty attribute. Further, the highest recognition accuracy attribute among the recognition accuracy attributes of all emotional types can be taken as the maximum recognition accuracy attribute. A first processing value is obtained by subtracting the maximum recognition accuracy attribute from a preset threshold value (such as 1); a cognitive uncertainty attribute is obtained by subtracting the recognition uncertainty attribute from the first processing value; and an emotional recognition confidence is obtained by subtracting the cognitive uncertainty attribute from the preset threshold value.

[0081] For example, the emotional recognition confidence can be determined based on formula (5). Formula (5) can be expressed as: A = 1 - u; u = 1 - max(p) + entropy(p); where u represents the cognitive uncertainty attribute; max(p) represents the maximum recognition accuracy attribute; the maximum recognition accuracy attribute is between 0 and 1; entropy(p) represents the recognition uncertainty attribute; and A represents the emotional recognition confidence. The recognition uncertainty attribute can be determined based on formula (6). Formula (6) can be expressed as: . represents the number of emotional types; represents the recognition accuracy attribute of the i-th emotional type; represents the recognition accuracy attribute of the i-th emotional type; represents a logarithmic function.

[0082] It should be noted that if the to-be-detected region is smaller than the preset area or the number of to-be-detected objects in the to-be-detected region is smaller than the preset value, it can be indicated that the area of the to-be-detected region is relatively small, or the objects gathered in the to-be-detected region are relatively few. At this time, the emotion recognition confidence corresponding to the same emotion type in the emotion data corresponding to different to-be-detected objects can be determined in combination with the recognition accuracy attribute corresponding to the same emotion type. The emotion recognition confidence corresponding to the emotion type includes: determining the recognition uncertainty attribute for the emotion type based on the recognition accuracy attribute corresponding to the same emotion type in the emotion data corresponding to different to-be-detected objects; determining the emotion recognition confidence of the emotion type based on the recognition accuracy attribute and the recognition uncertainty attribute corresponding to the emotion type. The emotion recognition confidence of each emotion type can be processed by mean value to obtain the final emotion recognition confidence.

[0083] S230, determining an emotion processing mode based on the object density heat map, the emotion type, and the emotion recognition confidence, to adjust the emotion of the to-be-detected object based on the emotion processing mode.

[0084] Among them, the emotion processing mode can be an intervention or response strategy taken according to the emotion recognition result and its confidence, used to guide, adjust or affect the emotional state of the to-be-detected object. Different emotion processing modes correspond to different interaction means, such as visual feedback, sound prompt, environment adjustment, etc.

[0085] In the embodiment, the object density heat map can reflect the object gathering density in the region to be detected, and if the density is high, it indicates that the crowd disorder is likely to occur, and if the density is small, it indicates that the number of people is small. The emotion recognition confidence can reflect the reliability of the current recognition result. If the confidence is high, it is considered that the emotion data is more trusted; if the confidence is low, it is considered that the emotion data is less trusted. The review mechanism can be triggered when the emotion recognition confidence is less than a first preset threshold (such as 0.6), for example, a more fine-grained local feature analysis subnetwork can be called to analyze the face image and the torso image to re-determine the emotion data of the object to be detected. If the emotion recognition confidence is greater than a second preset threshold, the emotion processing mode suitable for the current situation can be determined based on the emotion type corresponding to the maximum recognition accurate attribute in the emotion data, the emotion recognition confidence, and the object density heat map. For example, a plurality of processing rules can be determined in advance, such as: if the object density heat map shows that the density of the region is greater than a first preset density, the emotion recognition confidence is greater than a third preset threshold, and the emotion type of the maximum recognition accurate attribute is type a (such as anxiety), it is determined that the emotion processing mode is a first level response mode. The first level response mode can also include a plurality of modes, such as: adjusting the light to warm tone, guiding animation, notifying the nearest personnel (which can be determined by an indoor positioning system), adjusting the focus on the region to be detected, playing a customized soothing voice (which can be selected according to a custom language). If the object density heat map shows that the density of the region is greater than a second preset density, the emotion recognition confidence is greater than a fourth preset threshold, and the emotion type of the maximum recognition accurate attribute is type B (such as sadness), it is determined that the emotion processing mode is a second level response mode. If the object density heat map shows that the density of the region is greater than a third preset density, the emotion recognition confidence is greater than a fifth preset threshold, and the emotion type of the maximum recognition accurate attribute is type C, it is determined that the emotion processing mode is a third level response mode. After determining the emotion processing mode, active adjustment of the emotion of the object to be detected can be realized based on the emotion processing mode.

[0086] Exemplarily, the recognition accuracy attribute of each emotion type corresponding to the to-be-detected object can be included in the emotion data. In order to improve the reliability of the emotion processing mode decision, the uncertainty evaluation module based on the evidence theory can be used to calculate the cognitive uncertainty u = 1-max (p) + entropy (p), when u is greater than 0.4, the review mechanism is triggered, and a more fine local feature analysis subnetwork is called. The emotion recognition confidence is 1-u; the final system output includes the complete emotion analysis result of the emotion type, the emotion recognition confidence and the key detection area (visualized by Grad-CAM). In the scenic spot management end, these emotion analysis results are combined with the object density heat map to drive a three-level emergency response mechanism. For example, when a high emotion recognition confidence (such as an emotion recognition confidence greater than 90%) is recognized, the emotion type with the maximum recognition accuracy attribute is the angry type, and the area density is greater than 2 people per square meter, the emotion processing mode is automatically triggered to level 1 response. While realizing efficient and adaptive real environment emotion recognition, the intelligent level of scenic spot safety management is improved to meet the needs of actual emergency response.

[0087] On the basis of the above technical solutions, it should be noted that the technical solutions provided by the present application can be applied in any scene where emotion recognition of objects is required. The following is an example of different application scenarios. The present technical solution provides at least three specific deployment methods of systems for executing the present technical solution: one application scenario is a large-scale scenic spot central management system, which can use a cloud GPU cluster to run a complete model, support real-time analysis of up to 100 video streams, control the delay of each route within 500ms, and serve the tourist dispatch center and emergency command platform. Another application scenario is to deploy an edge computing node (such as an artificial intelligence device or an embedded system device in each subarea of the scenic spot), run a simplified version of the model (the number of channels is reduced to 3 / 4, and the input frame number is reduced to 12), process 8-10 video streams, and the delay is about 800ms, which is suitable for regional object emotion recognition. Another application scenario is a mobile terminal integrated into a handheld terminal for maintenance users, which processes a single video stream (the resolution is reduced to 640x480), uses a minimalist network obtained through knowledge distillation (only 1.5M parameters), realizes near real-time analysis (delay of 1.2s), and is used for on-site rapid disposal. The three methods can share the same core algorithm but adjust the network size and input configuration according to different hardware capabilities, and ensure the consistency of the overall system through cloud model cooperation.

[0088] The technical scheme provided in the embodiment generates an object density heat map based on a trunk image or a trunk feature map, so that the object density heat map can quantitatively reflect the body posture, action intensity and spatial distribution characteristics of a to-be-detected object, provide supplementary evidence at a behavior level for an emotional state, and enhance the perception ability of non-face emotional expression (such as limb stiffness when nervous and active action when excited). Further, the emotional recognition confidence is dynamically determined by analyzing the recognition accuracy attributes corresponding to different emotional types, so that the system can adaptively evaluate the reliability of the current recognition result, and avoid high-risk misjudgment. On this basis, the object activity level reflected by the object density heat map, the emotional type itself and the recognition confidence thereof are comprehensively considered, and a corresponding emotional processing mode (such as recommending soothing music, starting a psychological counseling conversation, adjusting environmental light or temperature, pushing a breathing training guide, etc.) is determined and matched, so as to realize personalized and dynamic emotional regulation intervention, not only improve the practicability and intelligence of emotional recognition, but also build a closed-loop control from emotional recognition to emotional intervention, and enhance the emotional support ability.

[0089] Figure 4 is a structural schematic diagram of an emotion recognition device provided according to an embodiment of the present application. As shown in the figure, the device comprises a data acquisition module 310, a normalization processing module 320, a feature map determination module 330 and an emotional data determination module 340. Figure 4

[0090] The data acquisition module 310 is configured to acquire a to-be-processed video including at least one to-be-detected object in a to-be-detected region and object feature data of the to-be-detected object. The to-be-processed video includes a plurality of video frames. The normalization processing module 320 is configured to perform spatio-temporal normalization processing on the plurality of video frames to obtain a face image and a trunk image corresponding to each video frame. The feature map determination module 330 is configured to input the face image into a trained face processing model to obtain a face feature map, and input the trunk image into a trained trunk processing model to obtain a trunk feature map. The emotional data determination module 340 is configured to, for each to-be-detected object, process the object feature data of the to-be-detected object to obtain object emotional features, and determine emotional data corresponding to the to-be-detected object based on the object emotional features, the face feature map and the trunk feature map.

[0091] ​The technical scheme of the embodiment comprises the following steps: acquiring a to-be-processed video comprising at least one to-be-detected object in a to-be-detected region and object feature data of the to-be-detected object; the to-be-processed video comprises a plurality of video frames; performing spatio-temporal normalization processing on the plurality of video frames to obtain a face image and a torso image corresponding to each video frame; inputting the face image into a trained face processing model to obtain a face feature map, and inputting the torso image into a trained torso processing model to obtain a torso feature map; for each to-be-detected object, processing the object feature data of the to-be-detected object to obtain object emotion features, and determining emotion data corresponding to the to-be-detected object based on the object emotion features, the face feature map, and the torso feature map. The emotion recognition in the prior art only depends on the recognition of object face features, and has the problem of low recognition accuracy. The embodiment realizes the spatio-temporal normalization processing on the plurality of video frames in the to-be-processed video, extracts standardized face images and torso images, inputs the face images and the torso images into the trained face processing model and the trained torso processing model respectively, obtains high-discriminative face feature maps and torso feature maps, processes the object feature data of the to-be-detected object to extract object emotion features, and then performs collaborative fusion analysis on the face and torso dynamic features in the visual mode and the non-visual object emotion features in the decision-making stage. The embodiment realizes the joint modeling of single mode to multi-dimensional and multi-source information, enhances the recognition ability of complex emotion states (such as suppressed anger and pseudo joy), improves the robustness in actual scenes such as low light, occlusion or long distance, improves the accuracy and reliability of emotion recognition, and effectively pacifies the to-be-detected object based on the emotion data.

[0092] On the basis of the above device, optionally, the normalization processing module 320 comprises:

[0093] A first image determination unit is configured to perform alignment processing on the plurality of video frames to obtain a plurality of first images.

[0094] A third image determination unit is configured to, for each first image, perform conversion processing on pixel color data in the first image to obtain a second image, and perform enhancement processing on the second image to obtain a third image.

[0095] A region detection unit is configured to detect the third image based on a region detection algorithm to obtain a face image and a torso image in the third image.

[0096] On the basis of the above device, optionally, the face processing model comprises a first convolutional layer, an attention module, a fully connected layer, and a second convolutional layer; and the feature map determination module 330 comprises:

[0097] A local picture feature determination unit is configured to sample the face image based on the first convolutional layer to obtain a local picture feature.

[0098] a channel descriptor determination unit configured to determine a channel descriptor based on processing of the local picture feature by the attention module;

[0099] a channel weight determination unit configured to determine a channel weight based on processing of the channel descriptor by the fully connected layer;

[0100] a face feature map determination unit configured to determine a face feature map based on processing of the face image by the channel weight.

[0101] In the above device, optionally, the face feature map determination unit comprises:

[0102] a weighted picture feature determination unit configured to determine a weighted picture feature based on weighting of the face image by the channel weight;

[0103] a second feature map determination unit configured to determine a first feature map based on max-pooling processing of the weighted picture feature, and determine a second feature map based on average-pooling processing of the weighted picture feature;

[0104] a spatial weight determination unit configured to determine a spatial weight based on processing of the first feature map and the second feature map by the second convolutional layer;

[0105] the face feature map determination unit is configured to determine the face feature map based on weighting of the weighted picture feature by the spatial weight.

[0106] In the above device, optionally, the emotion data determination module 340 comprises:

[0107] a to-be-fused feature data determination unit configured to determine to-be-fused feature data based on cross-attention processing of the face feature map and the torso feature map;

[0108] a multi-modal feature data determination unit configured to determine multi-modal feature data based on the to-be-fused feature data and the object emotion feature;

[0109] an emotion data determination unit configured to determine emotion data corresponding to the to-be-tested object based on the multi-modal feature data.

[0110] In the above device, optionally, the to-be-fused feature data determination unit comprises:

[0111] a cross-attention matrix determination unit configured to determine a cross-attention matrix based on processing of the face feature map and the torso feature map by a normalized exponential function;

[0112] The to-be-fused feature data determination unit is configured to determine to-be-fused feature data based on the trunk feature map, the face feature map, and the cross attention matrix.

[0113] On the basis of the above device, optionally, the device further comprises:

[0114] The object density heat map determination unit is configured to generate an object density heat map based on the trunk image and / or the trunk feature map.

[0115] The emotion recognition confidence determination unit is configured to determine an emotion recognition confidence based on the recognition accuracy properties corresponding to different emotion types in the emotion data corresponding to the to-be-tested object.

[0116] The emotion processing mode determination unit is configured to determine an emotion processing mode based on the object density heat map, the emotion type, and the emotion recognition confidence, so as to adjust the emotion of the to-be-tested object based on the emotion processing mode.

[0117] On the basis of the above device, optionally, the emotion recognition confidence determination unit comprises:

[0118] The recognition uncertainty property determination unit is configured to determine a recognition uncertainty property based on the recognition accuracy properties corresponding to different emotion types in the emotion data corresponding to the to-be-tested object.

[0119] The emotion recognition confidence determination unit is configured to determine the emotion recognition confidence based on the maximum recognition accuracy property and the recognition uncertainty property.

[0120] The emotion recognition device provided in the embodiments of the present application can execute the emotion recognition method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0121] Figure 5 is a structural schematic diagram of an electronic device for implementing the emotion recognition method in the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as a laptop computer, a desktop computer, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as a personal digital processing, a cellular phone, a smart phone, a wearable device (such as a helmet, glasses, a watch, etc.), and other similar computing devices. The components shown in the present document, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or claimed herein.

[0122] As Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory 12, a random access memory 13, etc., communicatively connected to the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory 12 or loaded into the random access memory 13 from the storage unit 18. Various programs and data required for the operation of the electronic device 10 can also be stored in the random access memory 13. The processor 11, the read-only memory 12, and the random access memory 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0124] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the emotion recognition method.

[0125] In some embodiments, the emotion recognition method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the read-only memory 12 and / or the communication unit 19. When the computer program is loaded into the random access memory 13 and executed by the processor 11, one or more steps of the emotion recognition method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the emotion recognition method by any other appropriate means, such as by means of firmware.

[0126] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip systems (SOCs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0127] Computer programs used to implement the processes of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0128] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory, read-only memory, erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0130] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0131] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0132] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs. For example, embodiments of the application include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program comprising program code for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the read only memory 12. When the computer program is executed by the processor 11, the above-described functions defined in the methods of the embodiments of the application are performed.

[0133] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the emotion recognition method provided by any of the embodiments of the present application.

[0134] The computer program product, in the implementation, can be written in one or more programming languages or combinations thereof to implement the computer program codes for performing the operation of the present application, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program codes can be executed completely on a user computer, partially on a user computer, as an independent software package, partially on a user computer and partially on a remote computer, or completely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet by using an Internet service provider).

[0135] It should be understood that the steps shown above can be reordered, added, or deleted, using various forms of flow. For example, the steps described in the present application can be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0136] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of emotion recognition, characterized by, The method comprises the following steps: acquiring a to-be-processed video including at least one to-be-detected object in a to-be-detected region and object feature data of the to-be-detected object; the to-be-processed video includes a plurality of video frames; performing spatio-temporal normalization processing on the plurality of video frames to obtain a face image and a torso image corresponding to each video frame; inputting the face image into a trained face processing model to obtain a face feature map, and inputting the torso image into a trained torso processing model to obtain a torso feature map; for each to-be-detected object, processing the object feature data of the to-be-detected object to obtain object emotion features, and determining emotion data corresponding to the to-be-detected object based on the object emotion features, the face feature map and the torso feature map.

2. The method of claim 1, wherein, The spatio-temporal normalization processing on the plurality of video frames to obtain a face image and a torso image corresponding to each video frame comprises: performing alignment processing on the plurality of video frames to obtain a plurality of first images; for each first image, performing conversion processing on pixel color data in the first image to obtain a second image, and performing enhancement processing on the second image to obtain a third image; detecting the third image based on a region detection algorithm to obtain a face image and a torso image in the third image.

3. The method of claim 1, wherein, The face processing model comprises a first convolutional layer, an attention module, a fully connected layer and a second convolutional layer; the inputting of the face image into the trained face processing model to obtain a face feature map comprises: sampling the face image based on the first convolutional layer to obtain local picture features; processing the local picture features based on the attention module to obtain channel descriptors; processing the channel descriptors based on the fully connected layer to obtain channel weights; processing the face image based on the channel weights to obtain a face feature map.

4. The method of claim 3, wherein, The processing of the face image based on the channel weights to obtain a face feature map comprises: weighting the face image according to the channel weights to obtain weighted picture features; performing maximum pooling processing on the weighted picture features to obtain a first feature map, and performing average pooling processing on the weighted picture features to obtain a second feature map; processing the first feature map and the second feature map based on the second convolutional layer to obtain spatial weights; weighting the weighted picture features according to the spatial weights to obtain a face feature map.

5. The method of claim 1, wherein, The determination of emotion data corresponding to the to-be-detected object based on the object emotion features, the face feature map and the torso feature map comprises: performing cross-attention processing on the face feature map and the torso feature map to obtain to-be-fused feature data; determining multi-modal feature data based on the to-be-fused feature data and the object emotion features; determining emotion data corresponding to the to-be-detected object based on the multi-modal feature data.

6. The method of claim 5, wherein, The cross-attention processing on the face feature map and the torso feature map to obtain to-be-fused feature data comprises: The face feature map and the torso feature map are processed based on a normalized exponential function to obtain a cross-attention matrix; Based on the torso feature map, the face feature map, and the cross-attention matrix, the to-be-fused feature data is determined.

7. The method of claim 1, wherein, The method further includes: generating an object density heat map based on the torso image and / or the torso feature map; determining an emotion recognition confidence based on an identification accuracy attribute corresponding to different emotion types in the emotion data corresponding to the to-be-tested object; determining an emotion processing mode based on the object density heat map, the emotion type, and the emotion recognition confidence, so as to adjust the emotion of the to-be-tested object based on the emotion processing mode.

8. The method of claim 1, wherein, The determination of the emotion recognition confidence based on the identification accuracy attribute corresponding to different emotion types in the emotion data corresponding to the to-be-tested object includes: determining an identification uncertainty attribute based on the identification accuracy attribute corresponding to different emotion types in the emotion data corresponding to the to-be-tested object; determining the emotion recognition confidence based on the maximum identification accuracy attribute and the identification uncertainty attribute.

9. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the emotion recognition method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute the emotion recognition method of any one of claims 1-8 when executed.

Images