A context-aware emotion recognition method, system, medium, and device

Abstract

The present disclosure provides a context-aware emotion recognition method, system, medium and device, relating to the technical field of image recognition, comprising extracting facial emotion information and human body contour information in a person image, obtaining facial emotion features and human body posture features; detecting the orientation and gaze information of the face, obtaining the positions of all objects in the object image, generating a depth map using a depth map generation algorithm, extracting the feature vector of the object, saving the boundary box coordinates, and constructing the interactive relationship of environmental objects in a three-dimensional perspective; extracting global scene context information features based on the background image; fusing the facial emotion features, human body posture features, three-dimensional perspective environmental object interaction relationship and global scene context relationship features to obtain new context emotion features, and inputting the new context emotion features into a model to obtain an emotion classification result; the present disclosure can improve the accuracy of context-aware emotion recognition.

Description

Technical Field

[0001] This disclosure relates to the field of image recognition technology, specifically to a three-dimensional context-aware emotion recognition method and system based on ViT (Vision Transformer) and Graph Convolutional Network (GCN). Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the development of deep learning technology, emotion recognition has become increasingly important in artificial intelligence research.

[0004] Emotion recognition technology is widely used in telemedicine, fatigue monitoring, and human-computer interaction systems. Early research on emotion recognition mainly focused on physiological signals (such as electroencephalogram (EEG), electrocardiogram (ECG), and electrooculogram (EOG), facial expressions, text, and speech. Facial expressions are generally considered the most natural way to express human emotions. However, due to complex environmental changes, facial information in some images inevitably becomes blurred. Therefore, emotion recognition systems based on facial expressions still require improvement.

[0005] As research into emotional information deepens, studies have shown that the surrounding environment and body posture play a crucial role in emotion perception. Visual scenes influence the actual structure of facial action encoding. Psychological research indicates that body language is an important way for humans to express emotional information. This provides a theoretical basis for context-aware emotion recognition methods. With the development of image acquisition devices, the resolution of captured images is increasing. In natural environments, information from the entire scene is often captured using these devices, providing a data foundation for context-aware emotion recognition methods. Furthermore, with the continuous improvement of hardware computing power, the efficiency of training deep learning models is also gradually increasing. In recent years, researchers have explored the application of deep learning technology in the field of emotion recognition, achieving high accuracy rates, especially in facial expression recognition using laboratory datasets. However, most facial expression recognition methods show poor accuracy in images captured in natural environments. This is mainly because images in natural environments often contain additional noise, affecting the performance of facial expression recognition systems. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes a context-aware emotion recognition method and system. It uses facial expressions, human posture, and the surrounding environment as the identifiable features for emotion recognition. Different algorithms are used for face detection. A method for constructing a 3DVG representation of the interaction between environmental objects is proposed, combining head posture and gaze angle to construct the influence of other objects in the environment on human emotions. An emotion feature fusion module is used to fuse and classify the emotion features.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] A context-aware emotion recognition method includes:

[0009] Acquire the scene image to be identified, extract the images of people, objects, and background from the scene image, and perform preprocessing;

[0010] Extract facial emotion information and human body contour information from human images to obtain facial emotion features and human posture features.

[0011] The algorithm detects the orientation and gaze information of human faces, obtains the position of all objects in the object image, generates a depth map using a depth map generation algorithm, extracts the feature vectors of objects, saves the bounding box coordinates, and constructs the interaction relationship between environmental objects in a 3D perspective; it also extracts global scene context information features based on the background image.

[0012] By fusing facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene contextual relationship features, new contextual emotion features are obtained and input into the model to obtain emotion classification results.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions:

[0014] A context-aware emotion recognition system includes:

[0015] The data acquisition module is used to acquire the scene image to be identified, extract the images of people, objects and background from the scene image, and perform preprocessing.

[0016] The facial expression feature extraction module and the human posture and emotion feature extraction module are used to extract facial emotion information and human contour information from human images, and obtain facial emotion features and human posture features.

[0017] The environmental object interaction relationship emotion feature extraction module is used to detect the orientation and gaze information of human faces, obtain the position of all objects in the object image, generate a depth map using a depth map generation algorithm, extract the feature vectors of objects, save the bounding box coordinates, and construct the environmental object interaction relationship from a three-dimensional perspective.

[0018] The global contextual emotion feature extraction module is used to extract global scene contextual information features based on the background image;

[0019] The emotion feature fusion and classification module is used to fuse facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene contextual relationship features to obtain new contextual emotion features, which are then input into the model to obtain emotion classification results.

[0020] According to some embodiments, the present disclosure adopts the following technical solutions:

[0021] A non-transitory computer-readable storage medium is provided for storing computer instructions that, when executed by a processor, implement a context-aware emotion recognition method as described above.

[0022] According to some embodiments, the present disclosure adopts the following technical solutions:

[0023] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform a context-aware emotion recognition method as described above.

[0024] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0025] This disclosure mainly comprises five parts: a data acquisition module, a facial expression feature extraction module, a human posture emotion feature extraction module, an environmental object interaction relationship emotion feature extraction module, a global context emotion feature extraction module, and an emotion feature fusion and classification module. Analysis reveals that in visual scenes, besides facial expressions, body posture and the surrounding environment (scene context) also reflect a person's emotional state. Therefore, this disclosure extracts these three types of information as discriminative features for emotion recognition. Openface is used for face detection to obtain the person's head posture and gaze angle. Based on the Faster-RCNN algorithm, object detection is performed. Combining head posture and gaze angle, a method for constructing a 3DVG representation of environmental object interaction relationships is proposed to model the influence of other objects in the environment on human emotions. Finally, the emotion feature fusion module fuses and classifies the emotion features.

[0026] This context-aware emotion recognition system, based on facial information, body posture information, environmental object interaction relationships, and global scene context information, fully extracts emotional information from images, laying a data foundation for the accuracy of emotion recognition. This disclosure uses ViT for image feature extraction and GNN to extract emotional information from the graphical representation of objects in the environment, laying a model foundation for the accuracy of emotion recognition. Furthermore, this disclosure employs a novel graphical model of environmental object interaction relationships based on depth maps, head posture, and gaze detection. This graphical model can effectively represent the influence of other objects in the environment on human emotions, improving the accuracy of context-aware emotion recognition. Attached Figure Description

[0027] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0028] Figure 1 This is a flowchart of a three-dimensional perspective context-aware emotion recognition method according to an embodiment of this disclosure;

[0029] Figure 2 This is a general structural diagram of the three-dimensional perspective context-aware emotion recognition method described in the embodiments of this disclosure. Detailed Implementation

[0030] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0032] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0033] Example 1

[0034] One embodiment of this disclosure provides a context-aware emotion recognition method, including:

[0035] Step 1: Obtain the scene image to be identified, extract the images of people, objects, and background from the scene image, and perform preprocessing;

[0036] Step 2: Extract facial emotion information and human body contour information from the image of the person to obtain facial emotion features and human posture features;

[0037] Step 3: Detect the orientation and gaze information of the face, obtain the position of all objects in the object image, generate a depth map using a depth map generation algorithm, extract the feature vectors of the objects, save the bounding box coordinates, and construct the interaction relationship between the environment objects in a 3D perspective; extract global scene context information features based on the background image.

[0038] Step 4: Fuse facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene contextual relationship features to obtain new contextual emotion features. Input these new contextual emotion features into the model to obtain the emotion classification result.

[0039] As one embodiment, in step 1, the preprocessing includes: converting the facial image in the person image to grayscale, performing face detection and face alignment operations, and then saving it.

[0040] In one embodiment, the facial emotion feature extraction method is as follows: extract facial images in the scene using a face detection algorithm, input them into a ViT model to extract the emotion information contained in the face, and obtain a feature vector of facial emotion information.

[0041] The method for extracting human posture features is as follows: extract human contour information using semantic segmentation algorithm, input the human contour information into ViT model to obtain feature vectors representing human posture and emotion.

[0042] The method for constructing the interaction relationship between environmental objects is as follows: detect the orientation and gaze information of the face, obtain the position of all objects in the object image through the object detection algorithm, generate the depth map of the image using the depth map generation algorithm, obtain the position of the objects in the image by combining the object detection algorithm, calculate the distance between each object and the camera, and then construct a graph representation of the interaction relationship between environmental objects. The graph structure representation is input into GCN to extract the emotional feature vector of the interaction relationship between environmental objects.

[0043] The method for extracting global scene context information features based on background images is as follows: supplement the global scene information ignored in the interaction relationship between environmental objects, focus on the influence of visual background on emotions, input the image that occludes the human body into the ViT network, and obtain the global context feature representation vector.

[0044] Then, the process of fusing facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene context features is as follows: The feature vectors representing facial emotions, human posture emotions, environmental object interaction relationships, and global context emotions are concatenated to obtain a new context-aware emotion feature vector, which is then input into the fully connected layer for image emotion classification. The specific implementation process is as follows:

[0045] For people in the image, OpenFace is used to extract their facial information, including complete facial images, gaze angles, and head pose information, to construct a 3D perspective graph model. Simultaneously, semantic segmentation algorithms are used to identify human contours to represent human pose information. A depth map generation algorithm is used to generate a depth map of the original image, estimating the distance from each pixel in the image to the camera center. Object detection algorithms are used to extract bounding boxes of various objects, retaining the feature vectors of each object obtained during this process, and saving the coordinates of their bounding boxes to construct environmental object interaction relationships. The background image refers to the portion of the image remaining after people and various objects are occluded; it can be used to extract global scene context information for emotion recognition.

[0046] The complete facial image undergoes preprocessing operations, including grayscale conversion, face detection, and face alignment, and the preprocessed image is saved. For the human contour information obtained using semantic segmentation algorithms, ViT is used to extract emotional information represented by human pose.

[0047] By combining various objects detected by object detection algorithms with depth maps, a connection between them and people is constructed, and a graph representation of environmental object relationships based on a 3D perspective is built. Emotional information from the graph representation is then learned using GCN.

[0048] For background images, the human body contours detected by semantic segmentation algorithms need to be occluded to prevent the model from learning human body information and affecting the extraction of environmental emotional information. The images are then input into ViT to extract global contextual features.

[0049] For four different emotional features, a feature fusion-based method is used to connect facial features, body posture features, graph representation features of environmental object relationships in a 3D perspective, and global contextual relationship features to obtain new contextual emotional features. A fully connected layer is added at the end of the model, and the contextual emotional feature vector is input into the fully connected layer to obtain the emotion classification result.

[0050] As one example, the method for constructing a graph representation of environmental object relationships based on a three-dimensional perspective is as follows:

[0051] Step S1: Faster-RCNN is used for object detection in image I, detecting the location of each object (obj) in the image. Let... This indicates the number of times Faster-RCNN detected the first... There are 10 objects, assuming Faster R-CNN detection results. The coordinates of the top left corner of the bounding box are The coordinates of the lower right corner are By setting the midpoint coordinates of this bounding box to... The location is calculated as follows:

[0052] (1)

[0053] in It is an image middle The location coordinates.

[0054] Step S2: Use the OpenFace toolbox to extract the gaze angle of the person in the image. The gaze angle extracted using OpenFace ( It consists of two parts, one of which represents the left and right perspectives. The other is to indicate the vertical perspective. Both of these variables are represented in radians. A 3D coordinate representation rule is established on the image: the image width direction is the positive x-axis, the image relative height direction is the positive y-axis, and the direction orthogonal to the two axes is the z-axis. A 3DVG (3D View Graph) is constructed based on these three coordinate axes. The first node defined in the 3DVG is its center node. That is, the human body node, using the coordinates of the center position of the human face as the coordinates of the center node. Features of the agent region extracted using Faster R-CNN This serves as a feature of the central node. Similarly, each object detected in the image is treated as a node in the 3DVG, i.e. When performing object detection on image regions, the convolutional modules of Faster-RCNN generate a vector feature for each object candidate region. When constructing 3DVG, as The node characteristics. The following section discusses the agent and each... Modeling the relationships between them, i.e., constructing the edges of a 3DVG.

[0055] As mentioned earlier, OpenFace extracts the perspective. It consists of two components, namely .in If the person's viewpoint is biased towards... flat The positive direction of the axis, otherwise negative. To and The axes are in the same direction. Similarly, if a person's viewpoint is biased towards... flat The positive direction of the axis, that is, looking down from above. If positive, otherwise negative, when the direction is... When the axes are the same Both of the above angles are points of view and... The angle along the positive direction of the axis. Additionally, it indicates... Direction vector Two corners can be used To calculate, the origin is... As The starting point. Let... exist The projection onto the plane is a unit vector. ,set up exist The projection on the plane is a vector ,from Calculate the coordinates of these two vectors.

[0056] (2)

[0057] (3)

[0058] (4)

[0059] in Representing vectors of Axis coordinates. Based on these two projection vectors, we can obtain the representation... Direction vector :

[0060] (5)

[0061] Step S3: Further construct the 3D direction vectors from the person to other objects in the image. The MegaDepth algorithm is used to generate a depth map, which represents the relative distance of each pixel in the image to the camera. The farther the relative distance of a pixel to the camera, the lower its pixel value in the depth map.

[0062] Let the image The depth map is When performing object detection on an image with the help of Faster-RCNN, each The bounding box of the object mentioned above. center coordinates The Euclidean distance is obtained using the centroid coordinates of the agent. Calculated, set as Next, use Constructing a 3D coordinate system Coordinates. First, {obj}_i shaft and The axis coordinates are respectively of shaft and Axis coordinates. Secondly, as mentioned before, Let pixel values ​​represent the distance from a pixel to the camera. middle The center pixel value ,but of The representation in the coordinate system is as follows:

[0063] (6)

[0064] in, It indicates that the center of the human being is The pixel value at that location, as shown in the above formula, is... When in front of people, It is positive if positive, otherwise negative. This assignment is also related to the gaze direction vector mentioned earlier. The construction process is the same. For ease of calculation, the plane where the person's coordinates are located is taken as... Plane, i.e. The axis value is 0, meaning the person's coordinates are... Calculating people and Euclidean distance in 3D coordinates :

[0065] (7)

[0066] in For a person in a two-dimensional coordinate system The Euclidean distance between them can be easily derived using the Pythagorean theorem, leading to the above formula. From the direction of the person... The vector can be represented as:

[0067] (8)

[0068] Step S4: In the graph structure and The relationships (i.e., edges) are represented by gaze direction vectors. , and Indicates. Due to and The smaller the angle between the two, the closer the relationship, meaning the more profound the influence of the object in a person's gaze on their emotions. Consider the cosine function in... The range is monotonous. and Border weight between The calculation formula is as follows:

[0069] (9)

[0070] (10)

[0071] in It is a vector and The cosine of the angle between them It is a weighting item, the purpose of which is to assign greater weight to objects that are closer to people when constructing the graph. This indicates the number of objects in the diagram.

[0072] Step S5: Construct a 3D VG based on the data obtained above, and use a graph convolutional neural network to learn the sentiment features in the graph. Let... ,in, express The node set in the image consists of objects detected in the image. express The set of edges in the middle, by composition; The eigenvector matrix representing the nodes contains There are n nodes, and the feature length of each node is 1. The graph structure built for each image. It is fed into a graph convolutional neural network to learn sentiment features. .

[0073] Furthermore, after extracting the four types of features, the personal emotional features (i.e., facial expression features and human posture emotional features) and scene context information (i.e., environmental object interaction relationship emotional features and global context emotional features) are concatenated to form the personal emotional features. and scene context emotional features In order to integrate and To suppress emotion-irrelevant information, a multiplicative fusion algorithm is introduced, the calculation formula of which is shown below:

[0074] (11)

[0075] in The total number of features considered is 2 in this invention. For the first The sentiment class provided by modal networks Predicting the type of emotion.

[0076] Furthermore, the trained context-aware emotion recognition system comprises five parts: a data acquisition module, a facial expression feature extraction module, a human posture emotion feature extraction module, an environmental object interaction relationship emotion feature extraction module, a global context emotion feature extraction module, and an emotion feature fusion module. The network structure is constructed using an image information extraction model based on ViT and a graph structure feature extraction model based on GCN.

[0077] Furthermore, training the model includes:

[0078] Step Q1: Set hyperparameters, including learning rate, batch size, dropout, and other hyperparameters;

[0079] Step Q2: Input the training data into the above modules for preprocessing, including face detection, human semantic segmentation, object detection, depth map generation, etc.

[0080] Step Q3: Establish a graph structure representation of the interaction relationships between environmental objects;

[0081] Step Q4: Construct a deep neural network model, including determining the number of layers in ViT, the number of layers in GCN, and constructing a fusion network.

[0082] Step Q5: Train the constructed multi-channel network model.

[0083] Example 2

[0084] One embodiment of this disclosure provides a context-aware emotion recognition system, including:

[0085] The data acquisition module is used to acquire the scene image to be identified, extract the images of people, objects and background from the scene image, and perform preprocessing.

[0086] The facial expression feature extraction module and the human posture and emotion feature extraction module are used to extract facial emotion information and human contour information from human images, and obtain facial emotion features and human posture features.

[0087] The environmental object interaction relationship emotion feature extraction module is used to detect the orientation and gaze information of human faces, obtain the position of all objects in the object image, generate a depth map using a depth map generation algorithm, extract the feature vectors of objects, save the bounding box coordinates, and construct the environmental object interaction relationship from a three-dimensional perspective.

[0088] The global contextual emotion feature extraction module is used to extract global scene contextual information features based on the background image;

[0089] The emotion feature fusion and classification module is used to fuse facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene contextual relationship features to obtain new contextual emotion features, which are then input into the model to obtain emotion classification results.

[0090] Example 3

[0091] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement a context-aware emotion recognition method as described above.

[0092] Example 4

[0093] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform a context-aware emotion recognition method as described above.

[0094] The steps and methods involved in the systems of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0095] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A context-aware emotion recognition method, characterized in that, include: Acquire the scene image to be identified, extract the images of people, objects, and background from the scene image, and perform preprocessing; Extract facial emotion information and human body contour information from human images to obtain facial emotion features and human posture features; among them, the ViT model is used to extract facial emotion features and human posture features. The process involves detecting the orientation and gaze information of faces, obtaining the positions of all objects in an image, generating a depth map using a depth map generation algorithm, extracting feature vectors of objects, saving bounding box coordinates, and constructing the interaction relationships between environmental objects from a 3D perspective. Specifically, this includes: Step S1: Detect the position of each object in the image; Step S2: Extract the left and right perspective angles of the person in the image. x , vertical perspective y Calculate the gaze direction vector based on the gaze angle. ; Step S3: Calculate the 3D vector v pointing from the person to the object based on the person's coordinates and the object's coordinates. i The x and y coordinates are determined based on the center position of the bounding box, and the z coordinate is determined based on the depth map generated by the MegaDepth algorithm, which represents the relative distance between each pixel in the image and the camera. Step S4: Construct a graph with the character and all objects as nodes, and calculate the gaze direction vector a. ga The vector v of the object pointed to by the person i The angle between the character and the object is calculated based on the character's coordinates and the object's coordinates. The edge weight is determined based on the angle and the Euclidean distance. The smaller the angle, the stronger the influence of the object on the character's emotions. The smaller the distance, the stronger the influence of the object on the character's emotions. Extracting global scene context information features based on background images; By fusing facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene contextual relationship features, new contextual emotion features are obtained and input into the model to obtain emotion classification results.

2. The context-aware emotion recognition method as described in claim 1, characterized in that, The preprocessing includes: converting the facial images in the person image to grayscale, performing face detection and face alignment operations, and then saving the image.

3. The context-aware emotion recognition method as described in claim 1, characterized in that, The method for extracting facial emotion features is as follows: facial images in the scene are extracted using a face detection algorithm, and then input into the ViT model to extract the emotion information contained in the face, thereby obtaining the feature vector of facial emotion information.

4. The context-aware emotion recognition method as described in claim 1, characterized in that, The method for extracting human posture features is as follows: extract human contour information using a semantic segmentation algorithm, and input the human contour information into the ViT model to obtain the feature vector representing the human posture emotion.

5. The context-aware emotion recognition method as described in claim 1, characterized in that, The method for constructing the interaction relationship between environmental objects is as follows: detect the orientation and gaze information of the face, obtain the position of all objects in the object image through the object detection algorithm, generate the depth map of the image using the depth map generation algorithm, obtain the position of the objects in the image by combining the object detection algorithm, calculate the distance between each object and the camera, then construct a graph representation of the interaction relationship between environmental objects, input the graph structure representation into GCN, and extract the emotional feature vector of the interaction relationship between environmental objects.

6. The context-aware emotion recognition method as described in claim 1, characterized in that, The method for extracting global scene context information features based on background images is as follows: supplement the global scene information ignored in the interaction relationship between environmental objects, focus on the influence of visual background on emotions, input the image that occludes the human body into the ViT network, and obtain the global context feature representation vector.

7. The context-aware emotion recognition method as described in claim 1, characterized in that, The process of fusing facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene context features is as follows: the feature vectors representing facial emotions, human posture emotions, environmental object interaction relationships, and global context emotions are concatenated to obtain a new context-aware emotion feature vector, which is then input into a fully connected layer for emotion classification of the image.

8. A context-aware emotion recognition system, characterized in that, The context-aware emotion recognition method according to any one of claims 1-7 includes: The data acquisition module is used to acquire the scene image to be identified, extract the images of people, objects and background from the scene image, and perform preprocessing. The facial expression feature extraction module and the human posture and emotion feature extraction module are used to extract facial emotion information and human contour information from human images, and obtain facial emotion features and human posture features. The environmental object interaction relationship emotion feature extraction module is used to detect the orientation and gaze information of human faces, obtain the position of all objects in the object image, generate a depth map using a depth map generation algorithm, extract the feature vectors of objects, save the bounding box coordinates, and construct the environmental object interaction relationship from a three-dimensional perspective. The global contextual emotion feature extraction module is used to extract global scene contextual information features based on the background image; The emotion feature fusion and classification module is used to fuse facial emotion features, human posture features, environmental object interaction relationships from a 3D perspective, and global scene contextual relationship features to obtain new contextual emotion features, which are then input into the model to obtain emotion classification results.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a context-aware emotion recognition method as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a context-aware emotion recognition method as described in any one of claims 1-7.

Images