Emotion recognition methods, devices, equipment and storage media

Abstract

This application discloses an emotion recognition method, apparatus, device, and storage medium. The application performs image segmentation on a portion of a user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is formed when the user's face is partially obscured by wearing a wearable device. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and complete facial contour segmentation maps of historical frames. Emotion recognition is then performed on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this application tracks full-face expressions in device-wearing scenarios using a pre-constructed image segmentation model and expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an emotion recognition method, apparatus, device, and storage medium. Background Technology

[0002] In Virtual Reality (VR), Augmented Reality (AR), and telecommunications, real-time, accurate facial expression tracking is crucial for creating lifelike virtual avatars. Existing technologies face two major challenges: occlusion issues: the frames or masks of head-mounted devices can obscure parts of the user's face (such as the forehead or around the eyes), causing traditional tracking algorithms based on the entire face to fail and thus unable to recognize user emotions. Summary of the Invention

[0003] The main purpose of this application is to provide an emotion recognition method, device, equipment, and storage medium, which aims to solve the technical problem that traditional emotion recognition methods cannot accurately identify user emotions when the face is occluded.

[0004] To achieve the above objectives, this application provides an emotion recognition method, which includes the following steps:

[0005] Based on a preset image segmentation model, a portion of the user's facial image is segmented to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is the image formed after the user wears the wearable device, excluding the part of the face that is obscured.

[0006] A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of the historical frames.

[0007] Based on a preset facial expression prediction model, the spatiotemporal sequence is used to identify emotions and obtain the user's emotion category.

[0008] Optionally, the step of performing emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category includes:

[0009] Based on a preset expression prediction model, the spatiotemporal sequence is used to predict expressions to obtain expression prediction vectors.

[0010] The emotion difference vector is determined based on the expression prediction vector and the general benchmark neutral vector.

[0011] Emotional categories are obtained by performing emotion recognition based on the emotion difference vector and the preset Voronoi diagram.

[0012] Optionally, the step of performing emotion recognition based on the emotion difference vector and a preset Voronoi diagram to obtain the emotion category includes:

[0013] Calculate the Euclidean distance between the emotion difference vector and the center point of the difference corresponding to each emotion category in the preset Voronoi diagram;

[0014] The nearest center point is determined from the difference center points based on the Euclidean distance and the nearest distance principle;

[0015] The emotion category is determined based on the emotion label associated with the cell corresponding to the nearest center point.

[0016] Optionally, before performing image segmentation on a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, the following steps are included:

[0017] Capture partial facial image samples of the wearer based on a preset average frame rate;

[0018] Emotion recognition is performed on the complete facial image sample of the wearer to obtain full-face facial feature parameters and emotion labels;

[0019] A preset training set is constructed based on the partial facial images, the full-face facial feature parameters, and emotion labels;

[0020] The initial image segmentation model is trained based on the preset training set to obtain the preset image segmentation model;

[0021] The initial expression prediction model is trained based on the preset training set to obtain the preset expression prediction model;

[0022] Full-face facial feature parameters labeled as neutral expression are extracted from the preset training set, and a preset Voronoi diagram is constructed based on the full-face facial feature parameters of the neutral expression.

[0023] Optionally, training the initial image segmentation model based on the preset training set to obtain the preset image segmentation model includes:

[0024] Randomly select a portion of face images from the preset training set at a preset ratio, and obtain an annotated facial contour segmentation map;

[0025] A pre-defined segmentation training set is constructed based on the partial face images and the labeled facial contour segmentation map;

[0026] The initial image segmentation model is trained based on the preset segmentation training set and the preset Dice loss function to obtain the preset image segmentation model. The initial image segmentation model uses ResNet18-FPN as the segmentation network architecture.

[0027] Optionally, training the initial expression prediction model based on the preset training set to obtain the preset expression prediction model includes:

[0028] The preset training set is labeled with active frames to obtain a target training set containing both active and inactive frames.

[0029] The initial expression prediction model is trained based on the target training set, preset weight factors, and preset weighted Huber loss function to obtain a preset expression prediction model. The initial prediction model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and then connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

[0030] Optionally, the step of extracting full-face facial feature parameters labeled as neutral expressions from the preset training set, and constructing a preset Voronoi diagram based on the full-face facial feature parameters of the neutral expressions, includes:

[0031] Extract full-face facial feature parameters labeled as neutral expression from the preset training set;

[0032] A general baseline neutral vector is calculated based on the full-face facial feature parameters of the neutral expression;

[0033] Calculate the emotion difference vector corresponding to each emotion category based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general benchmark neutral vector.

[0034] The center point of the difference for each emotion category is determined based on the mean of the emotion difference vectors corresponding to each emotion category.

[0035] A spatial index for the target dimension is constructed based on the difference center points corresponding to each emotion category, and a Voronoi diagram cell is constructed based on the spatial index and the emotion labels corresponding to the difference center points;

[0036] A preset Voronoi diagram is constructed based on the Voronoi diagram cells.

[0037] Furthermore, to achieve the above objectives, this application also provides an emotion recognition device, which includes:

[0038] The image segmentation module is used to segment a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is the image formed after the user wears the wearable device, excluding the part of the face that is obscured.

[0039] The sequence construction module is used to construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames.

[0040] The emotion recognition module is used to perform emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category.

[0041] In addition, to achieve the above objectives, this application also proposes an emotion recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the emotion recognition method described above.

[0042] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the emotion recognition method described above.

[0043] One or more technical solutions proposed in this application have at least the following technical effects:

[0044] This application performs image segmentation on a portion of a user's facial image based on a pre-defined image segmentation model to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is formed when the user's face is partially obscured by wearing a wearable device. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and complete facial contour segmentation maps of historical frames. An expression prediction model is then used to perform emotion recognition on the spatiotemporal sequence to obtain the user's emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this application tracks full-face expressions in device-wearing scenarios using a pre-constructed image segmentation model and expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the first embodiment of the emotion recognition method of this application;

[0048] Figure 2This is a flowchart illustrating the second embodiment of the emotion recognition method of this application.

[0049] Figure 3 This is a flowchart illustrating the third embodiment of the emotion recognition method of this application.

[0050] Figure 4 This is a schematic diagram of the overall scheme flow provided in the third embodiment of the emotion recognition method of this application;

[0051] Figure 5 This is a schematic diagram of the module structure of the emotion recognition device according to an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the emotion recognition method in the embodiments of this application.

[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0055] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of this application embodiment is as follows: This application performs image segmentation on a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is an image formed when the user wears a wearable device and partially obscures the face. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames. An emotion recognition is performed on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category.

[0057] In this embodiment, for ease of description, the following description uses a computing service device as the execution subject.

[0058] Traditional emotion recognition methods cannot accurately identify user emotions when the face is obscured.

[0059] This application provides a solution that tracks full-face expressions in device-wearing scenarios using a pre-built image segmentation model and an expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities.

[0060] As can be seen from the above embodiments, this application performs image segmentation on a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is formed when the user's face is partially obscured after wearing a wearable device. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames. An emotion recognition is then performed on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this application tracks full-face expressions in device-wearing scenarios using a pre-constructed image segmentation model and expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities.

[0061] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, including an emotion recognition system. The following description uses a computer as an example to illustrate this embodiment and the subsequent embodiments.

[0062] Based on this, the embodiments of this application provide an emotion recognition method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the emotion recognition method of this application.

[0063] In this embodiment, the emotion recognition method includes steps S10 to S30:

[0064] Step S10: Perform image segmentation on a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame.

[0065] It should be noted that the preset image segmentation model is a model trained based on ResNet18-FPN as the segmentation network architecture. This model is used to perform segmentation prediction processing on partial facial images to obtain a complete facial contour segmentation image. This model accurately extracts facial regions, providing clean input for subsequent processing. The preset image segmentation model is used to segment and recognize a portion of the user's facial image currently being processed by the user wearing a wearable device, obtaining a complete facial contour segmentation map for the current frame. Wearable devices can be devices that capture images of the wearer's face, such as smart glasses, VR headsets, and smart headphones. The complete facial contour segmentation map is a binary contour map containing only the facial skin region after removing all occlusions.

[0066] In its implementation, a real-time video stream is acquired via a camera on a wearable device. Each frame of the facial image in the video stream is then input into a pre-defined image segmentation model for segmentation prediction, resulting in a complete facial contour segmentation image. Therefore, the image segmentation model in this application can predict occluded areas, and the model output is a binary segmentation map. Pixels with a value of 1 correspond to the predicted facial skin region, while pixels with a value of 0 correspond to all non-facial regions, including glasses. In this way, the system achieves robustness against glasses occlusion, providing a standardized facial contour input unaffected by glasses for subsequent continuous blendshape (facial muscle movement) prediction.

[0067] Step S20: Construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames.

[0068] It should be noted that, in order to improve the accuracy of expression prediction, the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames are stacked to form a spatiotemporal sequence that reflects dynamic changes. The constructed spatiotemporal sequence is then input into a preset expression prediction model to recognize facial expressions. The complete facial contour segmentation map of historical frames refers to the segmentation map of the preceding N-1 frames stored in the circular buffer.

[0069] Step S30: Based on a preset expression prediction model, perform emotion recognition on the spatiotemporal sequence to obtain the user's emotion category.

[0070] It should be noted that the preset expression prediction model is a ResNet34_3D model obtained by improving the ResNet34 model and training the improved model. The ResNet34_3D model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and then connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

[0071] Understandably, this model is used to capture subtle, dynamic facial expression features, infer and predict these features, and output the facial expression prediction vector corresponding to the current frame. The facial expression prediction vector is a standardized 52-dimensional Blendshape facial motion vector.

[0072] In the specific implementation, the spatiotemporal sequence is input into the expression prediction model, which outputs a 52-dimensional facial motion vector corresponding to the current frame. Emotion recognition is then performed based on this 52-dimensional Blendshape facial motion vector to obtain the user's emotion category. The final output of 52 Blendshape parameters can be directly used to drive compatible virtual avatars (such as characters in Unity / Unreal Engine). Simultaneously, the output emotion category and intensity information can be used to trigger preset emotion animations for virtual avatars, adjust the dialogue strategy of the interaction system, or record the user's emotion change trends in mental health applications. All processing can be completed in real time on edge devices, meeting the interaction requirements of AR / VR applications.

[0073] In one possible implementation, this application can also rapidly personalize the system for new users. When a new user uses the system, a personalized adaptation process is initiated, including: a. Data collection: The user is prompted to perform approximately one minute of calibration, during which the user naturally makes neutral expressions and several basic expressions. The emotion recognition system simultaneously collects partial facial images. b. Model fine-tuning: Low-rank adaptive (LoRA) technology is used, adding a trainable rank decomposition matrix only to the last fully connected layer of the pre-trained Blendshape expression prediction model (ResNet34_3D), freezing all weights of the original model. The LoRA module is fine-tuned using data collected from the new user, making the prediction of full-face expressions more accurate. The introduction of the fine-tuning mechanism and personalized baseline vectors enables the system to quickly adapt to new users, providing individuals with a more accurate expression tracking and emotion recognition experience while maintaining universality.

[0074] This embodiment segments a portion of the user's facial image using a pre-defined image segmentation model to obtain a complete facial contour segmentation map of the current frame. The partial facial image is formed when the user's face is partially obscured by wearing a wearable device. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and complete facial contour segmentation maps of historical frames. An expression prediction model is then used to perform emotion recognition on the spatiotemporal sequence to obtain the user's emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this embodiment tracks full-face expressions in a device-wearing scenario using a pre-constructed image segmentation model and expression prediction model, achieving accurate emotion recognition. This effectively solves the tracking problem caused by lower facial occlusion and improves the cross-user robustness of emotion recognition through decoupled representation. It also supports efficient personalized fine-tuning and is suitable for scenarios such as real-time virtual avatar-driven and human-computer emotional interaction.

[0075] Based on the above Figure 1 The first embodiment shown illustrates a second embodiment of the emotion recognition method of this application; see also... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the emotion recognition method of this application. Based on the first embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description and will not be repeated hereafter.

[0076] In this embodiment, step S30 further includes:

[0077] Step S301: Based on a preset expression prediction model, perform expression prediction on the spatiotemporal sequence to obtain an expression prediction vector.

[0078] It should be noted that the trained ResNet34_3D model is used to infer and predict facial features, and outputs the facial prediction vector corresponding to the current frame. The facial prediction vector is a standardized 52-dimensional Blendshape facial motion vector B_pred.

[0079] Step S302: Determine the emotion difference vector based on the expression prediction vector and the general benchmark neutral vector.

[0080] It should be noted that the universal baseline neutral vector can be a pre-calculated neutral vector B_neutral based on the full-face facial feature parameters of a neutral expression. The emotion difference vector is determined by calculating the difference between the expression prediction vector and the universal baseline neutral vector.

[0081] Understandably, the formula for calculating the emotion difference vector of the current facial expression is as follows:

[0082] δ_pred = B_pred - B_neutral.

[0083] Step S303: Perform emotion recognition based on the emotion difference vector and the preset Voronoi diagram to obtain the emotion category.

[0084] Understandably, the calculated emotion difference vector of the current expression is mapped onto a preset Voronoi diagram, and emotion recognition is performed based on the emotion difference vector and each emotion center point in the preset Voronoi diagram to obtain the emotion category.

[0085] It should be understood that the preset Voronoi diagram is a pre-set Voronoi diagram used to realize emotion categories. This Voronoi diagram is a spatial index structure for the target dimension built based on the full-face facial features of neutral expressions and the full-face facial feature parameters corresponding to all emotions. A general baseline neutral vector is calculated through the full-face facial feature parameters of neutral expressions. Based on the general baseline neutral vector and the full-face facial feature parameters corresponding to all emotions, the emotion difference vector corresponding to each emotion category is calculated. The difference center point corresponding to each emotion category is determined based on the emotion difference vector. Thus, a spatial index structure for the target dimension is constructed based on the difference center point. The Voronoi diagram cells are constructed based on the spatial index result and the difference center point to generate the Voronoi diagram.

[0086] Furthermore, step S303 further includes: calculating the Euclidean distance between the emotion difference vector and the difference center points corresponding to each emotion category in the preset Voronoi diagram; determining the nearest center point from the difference center points according to the Euclidean distance and the nearest distance principle; and determining the emotion category according to the emotion label associated with the cell corresponding to the nearest center point.

[0087] It should be noted that the Voronoi diagram transforms the complex classification problem into a nearest neighbor query in a high-dimensional space. The decision-making process is transparent, and the cells can intuitively display the distribution relationship of different emotions in the Blendshape difference space. In the process of querying and classifying emotions based on the Voronoi diagram, the emotion difference vector δ_pred is mapped to a preloaded Voronoi diagram. The Euclidean distance between the emotion difference vector δ_pred and the difference center point {C_emotion} corresponding to each emotion category is calculated, and the nearest center point is selected from among all center points based on the principle of proximity. The emotion difference vector δ_pred is classified into the Voronoi cell corresponding to this center point, and the emotion label associated with this cell is the dominant emotion identified by the system. In specific implementation, this application can also achieve emotion intensity estimation by simultaneously calculating the L2 norm (modulus) of δ_pred as a primary indicator of emotion intensity. The larger the norm, the greater the amplitude of facial expression changes, and the stronger the emotion may be. More precisely, the relative distance from δ_pred to the center of its assigned cell can be calculated.

[0088] This embodiment segments a portion of a user's facial image using a pre-defined image segmentation model to obtain a complete facial contour segmentation map of the current frame. The partial facial image is formed when the user's face is partially obscured by wearing a wearable device. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and complete facial contour segmentation maps of historical frames. An expression prediction vector is obtained by predicting expressions based on the spatiotemporal sequence using a pre-defined expression prediction model. An emotion difference vector is determined based on the expression prediction vector and a general baseline neutral vector. Emotion recognition is then performed based on the emotion difference vector and a pre-defined Voronoi diagram to obtain the emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this embodiment tracks full-face expressions in device-wearing scenarios using a pre-constructed image segmentation model and expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities.

[0089] Based on the above Figure 1 The first embodiment shown is followed by a third embodiment of the emotion recognition method of this application; see also... Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the emotion recognition method of this application. Based on the first embodiment of this application, in the third embodiment, the content that is the same as or similar to the first embodiment described above can be referred to the above description and will not be repeated hereafter.

[0090] In this embodiment, the method further includes the following step before step S10:

[0091] Step S01: Capture partial facial image samples of the user based on a preset average frame rate.

[0092] It should be noted that during the construction of the training set, dataset collection can be completed based on a pre-built data acquisition system. This system can include three hardware components: a smart wearable device, such as smart glasses (containing dual low-power cameras), a smart terminal, such as a smartphone (running a face capture application like Live Link Face), and a PC as a control center. The preset average frame rate is a pre-set statistical average of the number of image frames captured per second by the smart wearable device's cameras over a period of time. This frame rate can be adjusted according to the performance of the smart wearable device.

[0093] In practice, the data acquisition process is uniformly scheduled through a PC-based control program. When acquisition begins, the glasses' camera captures facial images of the wearer's face, excluding the obscured areas, at a preset average frame rate (e.g., 30fps) as partial facial image samples.

[0094] Step S02: Perform emotion recognition on the complete facial image sample of the wearer to obtain full-face facial feature parameters and emotion labels.

[0095] Understandably, while capturing partial facial images of the wearer through the smart wearable device's camera, the smart terminal's camera simultaneously captures the wearer's complete facial image, obtaining a complete facial image sample. The corresponding expression recognition application on the smart terminal then performs emotion recognition on the complete facial image sample, obtaining full-face facial feature parameters and emotion labels. Data alignment is achieved through timestamps and network protocols. The complete facial image refers to the facial image of the user while wearing the smart wearable device.

[0096] While capturing partial facial images of the wearer, the system simultaneously controls the smartphone's front-facing camera to take a full frontal shot of the same user. The system then uses the smartphone's built-in Live Link Face application to analyze and output 52 Blendshape parameters as real-world facial expression data in real time. Each frame is labeled with a corresponding emotion tag. These emotion tags can include joy, sadness, anger, surprise, disgust, fear, and neutrality.

[0097] Step S03: Construct a preset training set based on the partial facial images, the full-face facial feature parameters, and the emotion labels.

[0098] It should be noted that a preset training set is constructed by acquiring partial facial images, the full-face facial feature parameters, and emotion labels.

[0099] Step S04: Train the initial image segmentation model based on the preset training set to obtain the preset image segmentation model.

[0100] It should be noted that the initial image segmentation model uses ResNet18-FPN as the segmentation network architecture. The initial image segmentation model is trained using a pre-defined training set to obtain the pre-defined image segmentation model.

[0101] Furthermore, to illustrate the segmentation model training process, step S04 further includes: randomly selecting a portion of facial images of a preset proportion from the preset training set, and obtaining an annotated facial contour segmentation map; constructing a preset segmentation training set based on the portion of facial images and the annotated facial contour segmentation map; training an initial image segmentation model based on the preset segmentation training set and a preset Dice loss function to obtain a preset image segmentation model, wherein the initial image segmentation model uses ResNet18-FPN as the segmentation network architecture.

[0102] It should be noted that a portion of facial images of a preset proportion are randomly selected from the preset training set. This preset proportion can be set according to training needs, such as 5%. The preset Dice loss function is a pre-set metric used for fitting and statistically analyzing the pixel set. This application determines the number of pixels in the overlapping part of the predicted and real faces, as well as the sum of the total number of pixels in the predicted and real faces, by fitting the predicted and real face pixel set output by the model. The Dice coefficient is obtained by dividing twice the number of pixels in the overlapping part of the predicted and real faces by the sum of the total number of pixels in the predicted and real faces. The formula for calculating the Dice coefficient is as follows:

[0103] ;

[0104] Where P: the set of pixels predicted by the model as a face. G: the set of pixels of the real face. |P∩G|: the number of pixels overlapping between the predicted and real faces (intersection). |P| + |G|: the sum of the total number of pixels of the predicted face and the real face. The Dice coefficient ranges from [0,1], where 1 represents a perfect match, completely overlapping. 0 indicates no overlap at all. Since a larger Dice coefficient is better, while a smaller loss function is better, the Dice loss is usually defined as: L Dice =1 Dice(P,G) is a minimizeable Dice loss function obtained by maximizing the overlap between the prediction and the true label.

[0105] Understandably, approximately 5% of facial images are randomly selected from a pre-defined training set, and facial contour segmentation maps are manually labeled to construct a segmentation training set for training a model that can predict complete facial contours from partial images. ResNet18-FPN is used as the segmentation network architecture, with a 320×240 pixel partial facial image as input and a binary segmentation map of the same size as output (face region = 1, background = 0). The Dice loss function is used for training until the model can accurately predict complete facial contours from occluded images.

[0106] Step S05: Train the initial expression prediction model based on the preset training set to obtain the preset expression prediction model.

[0107] It should be noted that the initial expression prediction model is trained using a preset training set to obtain the preset expression prediction model. The initial expression model is an improved version of the ResNet34 model, and the preset expression prediction model is a model trained using the improved ResNet34 model.

[0108] Furthermore, to illustrate the training process of the preset expression prediction model, step S05 further includes: labeling the preset training set with active frames to obtain a target training set containing active and inactive frames; training the initial expression prediction model based on the target training set, preset weight factors, and preset weighted Huber loss function to obtain the preset expression prediction model. The initial prediction model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and sequentially connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

[0109] It should be noted that the expression prediction model is trained using a preset training set. This is achieved by inputting the complete training set into the trained segmentation model and outputting a predicted segmentation map. The segmentation image of the current frame is stacked with the segmentation images of the previous N frames in chronological order to form a spatiotemporal sequence of the target size. This spatiotemporal sequence is then input into the initial expression prediction model for training. During training, a weighted Huber loss function is used to balance the regression difficulty between active expression frames and neutral frames.

[0110] Understandably, for each sample in the pre-defined training set, its emotion label is used to determine whether it is an active frame. Samples with a "neutral" emotion label are labeled as inactive frames, while samples with other emotion labels (such as happy, sad, etc.) are labeled as active frames. The samples labeled with active / inactive frames are then integrated with the original data (including partial facial images, complete facial images, ground truth values ​​of Blendshape parameters, and emotion labels) to form the target training set. Based on the target training set and the pre-defined weighted Huber loss function, the initial expression prediction model is trained to obtain the pre-defined expression prediction model.

[0111] It should be understood that, by pre-setting weight factors, two weight factors are calculated based on the proportion of active and inactive frames in the target training set: f n (Active frame weights) and f a (Inactive frame weight). Specific calculation method: Let the number of active frames be N. n The number of inactive frames is N a If the number of emotion categories is K (excluding neutral), then ; These weights will be used to weight the Huber loss function to balance the contributions of active and inactive frames to the loss function. The Huber loss function combines the mean error (MSE) and the mean absolute error (MAE), and its formula is as follows:

[0112] ;

[0113] Here, δ is a hyperparameter that controls the switching point from MSE to MAE.

[0114] The pre-defined weighted Huber loss function is:

[0115] ;

[0116] Where B is the batch size, i n (y) is an indicator function, which is 1 when sample i is an inactive frame. y is the true value, where i is the actual value. a Similarly, f(x) is the model's predicted value. These are the weights of the facial expressions in the active frames, where K is the number of facial expression types in the dataset. It is the number of active frames. It is the number of inactive frames. This indicates that the current frame is the active frame. The weights are for inactive frames, i.e., frames with natural facial expressions. This indicates that the current frame is inactive. 0) is a hyperparameter used to control the switching point of Huber loss between MSE and MAE. It occurs when the absolute value of the error between the predicted and actual values ​​is less than or equal to... When the absolute value of the error is greater than 1, use mean squared error; when the absolute value of the error is greater than 1. When using linear functions, the impact of outliers on the loss can be reduced.

[0117] The ResNet34_3D model is initialized by replacing 2D convolutional layers with 3D convolutional layers and batch normalization layers with 3D batch normalization layers. A flattening layer and a fully connected layer are then added to the network, outputting a 52-dimensional vector. Data from the target training set is organized into batches, each batch containing B samples. Each sample is a segmentation map of N consecutive frames (obtained from the first-stage segmentation model) and the corresponding ground truth 52-dimensional Blendshape parameters. For each batch: a. Forward propagation: The N-frame segmentation map sequence is input into the model to obtain a 52-dimensional Blendshape prediction vector. b. Loss calculation: The weighted Huber loss is calculated based on the ground truth Blendshape parameters and the predicted values. c. Backpropagation: The gradient is calculated and the model parameters are updated. This process is repeated for multiple training epochs until the model converges, obtaining the preset facial expression prediction model.

[0118] In the specific implementation, the glasses image is input into a pre-trained segmentation model to obtain a predicted segmentation map. This segmentation map is then stacked chronologically with the segmentation maps of the preceding N frames (e.g., N=4) to form a spatiotemporal input tensor of size N×320×240. A 3D convolutional network based on ResNet34 (ResNet34_3D) is constructed: all 2D convolutional layers in the original ResNet34 are replaced with corresponding 3D convolutional layers, and batch normalization layers are replaced with 3D batch normalization layers. The network is finally connected to flattening layers and fully connected layers, outputting a 52-dimensional vector. When the spatiotemporal sequence is input into the ResNet34_3D network, the network outputs a 52-dimensional blendshape predicted value.

[0119] Step S06: Extract full-face facial feature parameters labeled as neutral expression from the preset training set, and construct a preset Voronoi diagram based on the full-face facial feature parameters of the neutral expression.

[0120] It should be noted that in the Voronoi emotion difference space construction stage of this application, after the model training is completed, a Voronoi space for emotion recognition is constructed by extracting full-face facial feature parameters with neutral emotion labels from the preset training set, and constructing a preset Voronoi map based on the full-face facial feature parameters of neutral emotions.

[0121] In the specific implementation, to further illustrate the training construction and online inference process after the construction is completed, refer to... Figure 4The schematic diagram of the overall solution shows that data acquisition and synchronous acquisition of partially occluded facial images captured by a wearable device, as well as ground truth values ​​of full-face blendshape parameters and emotion labels generated by an independent facial capture system, are used to train a two-stage neural network. The first stage uses a ResNet18-FPN network to segment facial contours based on partial facial images. The second stage uses a ResNet34 model network to replace the 3D convolutional layers and predicts the 52-dimensional blendshape parameters of the full face based on facial contours from multiple consecutive frames. Based on the ground truth values ​​of blendshape in the training data, a general baseline neutral blendshape vector is calculated. The difference between the blendshape vector of each emotion sample and the baseline vector is calculated. A Voronoi diagram centered on the emotion category is constructed based on the difference vector. For the input partial facial images, the two-stage network is used sequentially to obtain the predicted blendshape vector. The difference between the predicted vector and the baseline vector is calculated. The difference vector is mapped to the Voronoi diagram, and the emotion category is determined based on the Voronoi cell it falls into. A general baseline neutral vector is calculated based on the full-face facial feature parameters of neutral expressions. For each emotion category in the training set, the mean of all corresponding difference vectors is calculated based on the general baseline neutral vector to obtain the center point of that emotion category. A spatial index structure is then constructed based on this center point. This spatial index structure and the emotion labels corresponding to each center point are serialized and stored as a data file, thus obtaining the preset Voronoi diagram.

[0122] Further, step S06 also includes: extracting full-face facial feature parameters of neutral expression from the preset training set; calculating a general baseline neutral vector based on the full-face facial feature parameters of the neutral expression; calculating the emotion difference vector corresponding to each emotion category based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general baseline neutral vector; determining the difference center point corresponding to each emotion category based on the mean of the emotion difference vectors corresponding to each emotion category; constructing a spatial index of the target dimension based on the difference center points corresponding to each emotion category, and constructing Voronoi diagram cells based on the spatial index and the emotion labels corresponding to the difference center points; and constructing a preset Voronoi diagram based on the Voronoi diagram cells.

[0123] It should be noted that, from the preset training set, full-face facial feature parameters with neutral emotion labels are extracted to calculate a general baseline neutral vector. Based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general baseline neutral vector, the emotion difference vector corresponding to each emotion category is calculated. The mean of the emotion difference vectors corresponding to each emotion category is used to determine the difference center point corresponding to each emotion category. The difference center point corresponding to each emotion category is used as the generation point to construct a spatial index for the target dimension. Based on the spatial index and the emotion label corresponding to the difference center point, a Voronoi diagram cell is constructed. Based on the Voronoi diagram cell, a preset Voronoi diagram is constructed.

[0124] In the specific implementation, the Voronoi space construction process is as follows: a. Calculate the general baseline neutral vector: Extract the true Blendshape values ​​corresponding to all samples labeled as "neutral" expressions from the training dataset. Calculate the arithmetic mean of these vectors to obtain a 52-dimensional general baseline neutral Blendshape vector, denoted as B_neutral. b. Calculate the emotion difference vector: For each sample with an emotion label in the dataset, obtain its Blendshape vector B_i, and calculate its emotion difference vector δ_i = B_i - B_neutral. This operation aims to strip away individual static facial features and obtain the amount of expression change purely driven by emotion. c. Calculate the emotion category centroid: Group all samples according to their emotion labels. For each basic emotion category (such as happiness, sadness, anger, surprise, etc.), calculate the mean of the difference vector δ_i of all samples belonging to that category to obtain the difference centroid C_emotion for that emotion. d. Constructing the Voronoi diagram: The difference center points {C_happy, C_sad, C_angry, ...} across all emotion categories are used as seed points. In the 52-dimensional difference vector space, the Voronoi partitioning algorithm is applied: any point in the space belongs to the region represented by its nearest seed point. This generates a Voronoi diagram, where each convex polygon region (Voronoi cell) corresponds to an emotion category. This diagram structure is serialized and saved for subsequent inference.

[0125] This embodiment captures partial facial image samples of the user based on a preset average frame rate; performs emotion recognition on the complete facial image samples of the user to obtain full-face facial feature parameters and emotion labels; constructs a preset training set based on partial facial images, full-face facial feature parameters, and emotion labels; trains an initial image segmentation model based on the preset training set to obtain a preset image segmentation model; trains an initial expression prediction model based on the preset training set to obtain a preset expression prediction model; extracts full-face facial feature parameters with neutral emotion labels from the preset training set, and constructs a preset Voronoi diagram based on the full-face facial feature parameters of neutral expressions; performs image segmentation on partial facial images of the user based on the preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, wherein partial facial images are images formed when the user's face is partially obscured after wearing the wearable device; constructs a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames; performs emotion recognition on the spatiotemporal sequence based on the preset expression prediction model to obtain the user's emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this embodiment tracks full-face expressions in device-wearing scenarios using a pre-built image segmentation model and expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities.

[0126] This application also provides an emotion recognition device; please refer to [reference needed]. Figure 5 The emotion recognition device includes:

[0127] Image segmentation module 10 is used to segment a portion of a user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, wherein the portion of the facial image is an image formed after the user wears a wearable device, excluding the part of the face that is obscured.

[0128] Sequence construction module 20 is used to construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation map of the historical frames;

[0129] The emotion recognition module 30 is used to perform emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category.

[0130] Furthermore, the emotion recognition module 30 is also used to predict the expression of the spatiotemporal sequence based on a preset expression prediction model to obtain an expression prediction vector; determine an emotion difference vector based on the expression prediction vector and a general benchmark neutral vector; and perform emotion recognition based on the emotion difference vector and a preset Voronoi diagram to obtain an emotion category.

[0131] Furthermore, the emotion recognition module 30 is also used to calculate the Euclidean distance between the emotion difference vector and the difference center point corresponding to each emotion category in the preset Voronoi diagram; determine the nearest center point from the difference center points according to the Euclidean distance and the nearest distance principle; and determine the emotion category according to the emotion label associated with the cell corresponding to the nearest center point.

[0132] Furthermore, the emotion recognition device also includes a model building module, which is used to capture partial facial image samples of the wearer based on a preset average frame rate; perform emotion recognition on the complete facial image samples of the wearer to obtain full-face facial feature parameters and emotion labels; construct a preset training set based on the partial facial images, the full-face facial feature parameters, and the emotion labels; train an initial image segmentation model based on the preset training set to obtain a preset image segmentation model; train an initial expression prediction model based on the preset training set to obtain a preset expression prediction model; extract full-face facial feature parameters with neutral emotion labels from the preset training set, and construct a preset Voronoi diagram based on the full-face facial feature parameters of neutral expressions.

[0133] Furthermore, the model building module is also used to randomly select a portion of face images of a preset proportion from the preset training set and obtain an annotated facial contour segmentation map; construct a preset segmentation training set based on the portion of face images and the annotated facial contour segmentation map; train the initial image segmentation model based on the preset segmentation training set and the preset Dice loss function to obtain a preset image segmentation model, wherein the initial image segmentation model adopts ResNet18-FPN as the segmentation network architecture.

[0134] Furthermore, the model building module is also used to annotate the preset training set with active frames to obtain a target training set containing active and inactive frames; and to train the initial expression prediction model based on the target training set, preset weight factors, and preset weighted Huber loss function to obtain a preset expression prediction model. The initial prediction model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and then sequentially connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

[0135] Furthermore, the model building module is also used to extract full-face facial feature parameters with neutral emotion labels from the preset training set; calculate a general baseline neutral vector based on the full-face facial feature parameters of the neutral expression; calculate the emotion difference vector corresponding to each emotion category based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general baseline neutral vector; determine the difference center point corresponding to each emotion category based on the mean of the emotion difference vectors corresponding to each emotion category; construct a spatial index of the target dimension based on the difference center points corresponding to each emotion category, and construct Voronoi diagram cells based on the spatial index and the emotion labels corresponding to the difference center points; and construct a preset Voronoi diagram based on the Voronoi diagram cells.

[0136] The emotion recognition device provided in this application, employing the emotion recognition method described in the above embodiments, can solve the technical problem that traditional emotion recognition methods cannot accurately identify user emotions when the face is obscured. Compared with the prior art, the beneficial effects of the emotion recognition device provided in this application are the same as those of the emotion recognition method provided in the above embodiments, and other technical features in the emotion recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0137] This application provides an emotion recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the emotion recognition method in the first embodiment described above.

[0138] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the emotion recognition device in the embodiments of this application. The emotion recognition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The emotion recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0139] like Figure 6As shown, the emotion recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the emotion recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the emotion recognition device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows emotion recognition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0140] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0141] The emotion recognition device provided in this application, employing the emotion recognition method described in the above embodiments, can solve the technical problem that traditional emotion recognition methods cannot accurately identify user emotions when the face is obscured. Compared with the prior art, the beneficial effects of the emotion recognition device provided in this application are the same as those of the emotion recognition method provided in the above embodiments, and other technical features of this emotion recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0143] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0144] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the emotion recognition method in the above embodiments.

[0145] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0146] The aforementioned computer-readable storage medium may be included in the emotion recognition device; or it may exist independently and not incorporated into the emotion recognition device.

[0147] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the emotion recognition device, cause the emotion recognition device to: perform image segmentation on a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, wherein the portion of the facial image is an image formed after the user wears a wearable device, excluding the part of the face that is obscured; construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames; and perform emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category.

[0148] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0151] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described emotion recognition method, thereby solving the technical problem of emotion recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the emotion recognition method provided in the above embodiments, and will not be repeated here.

[0152] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An emotion recognition method, characterized in that, The method includes: Based on a preset image segmentation model, a portion of the user's facial image is segmented to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is the image formed after the user wears the wearable device, excluding the part of the face that is obscured. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of the historical frames. Based on a preset facial expression prediction model, emotion recognition is performed on the spatiotemporal sequence to obtain the user's emotion category; The step of performing emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category includes: Based on a preset expression prediction model, the spatiotemporal sequence is used to predict expressions to obtain expression prediction vectors. The emotion difference vector is determined based on the expression prediction vector and the general benchmark neutral vector. Calculate the Euclidean distance between the emotion difference vector and the center point of the difference corresponding to each emotion category in the preset Voronoi diagram; The nearest center point is determined from the difference center points based on the Euclidean distance and the nearest distance principle; The emotion category is determined based on the emotion label associated with the cell corresponding to the nearest center point.

2. The emotion recognition method as described in claim 1, characterized in that, Before performing image segmentation on a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, the process includes: Capture partial facial image samples of the wearer based on a preset average frame rate; Emotion recognition is performed on the complete facial image sample of the wearer to obtain full-face facial feature parameters and emotion labels; A preset training set is constructed based on the partial facial images, the full-face facial feature parameters, and emotion labels; The initial image segmentation model is trained based on the preset training set to obtain the preset image segmentation model; The initial expression prediction model is trained based on the preset training set to obtain the preset expression prediction model; Full-face facial feature parameters labeled as neutral expression are extracted from the preset training set, and a preset Voronoi diagram is constructed based on the full-face facial feature parameters of the neutral expression.

3. The emotion recognition method as described in claim 2, characterized in that, The step of training the initial image segmentation model based on the preset training set to obtain the preset image segmentation model includes: Randomly select a portion of face images from the preset training set at a preset ratio, and obtain an annotated facial contour segmentation map; A pre-defined segmentation training set is constructed based on the partial face images and the labeled facial contour segmentation map; The initial image segmentation model is trained based on the preset segmentation training set and the preset Dice loss function to obtain the preset image segmentation model. The initial image segmentation model uses ResNet18-FPN as the segmentation network architecture.

4. The emotion recognition method as described in claim 3, characterized in that, The step of training the initial expression prediction model based on the preset training set to obtain the preset expression prediction model includes: The preset training set is labeled with active frames to obtain a target training set containing both active and inactive frames. The initial expression prediction model is trained based on the target training set, preset weight factors, and preset weighted Huber loss function to obtain a preset expression prediction model. The initial prediction model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and then connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

5. The emotion recognition method as described in claim 2, characterized in that, The step of extracting full-face facial feature parameters labeled as neutral expressions from the preset training set, and constructing a preset Voronoi diagram based on the full-face facial feature parameters of the neutral expressions, includes: Extract full-face facial feature parameters labeled as neutral expression from the preset training set; A general baseline neutral vector is calculated based on the full-face facial feature parameters of the neutral expression; Calculate the emotion difference vector corresponding to each emotion category based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general benchmark neutral vector. The center point of the difference for each emotion category is determined based on the mean of the emotion difference vectors corresponding to each emotion category. A spatial index for the target dimension is constructed based on the difference center points corresponding to each emotion category, and a Voronoi diagram cell is constructed based on the spatial index and the emotion labels corresponding to the difference center points; A preset Voronoi diagram is constructed based on the Voronoi diagram cells.

6. An emotion recognition device, characterized in that, The device includes: The image segmentation module is used to segment a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is the image formed after the user wears the wearable device, excluding the part of the face that is obscured. The sequence construction module is used to construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames. The emotion recognition module is used to perform emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category; The step of performing emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category includes: Based on a preset expression prediction model, the spatiotemporal sequence is used to predict expressions to obtain expression prediction vectors. The emotion difference vector is determined based on the expression prediction vector and the general benchmark neutral vector. Calculate the Euclidean distance between the emotion difference vector and the center point of the difference corresponding to each emotion category in the preset Voronoi diagram; The nearest center point is determined from the difference center points based on the Euclidean distance and the nearest distance principle; The emotion category is determined based on the emotion label associated with the cell corresponding to the nearest center point.

7. An emotion recognition device, characterized in that, The emotion recognition device includes: a memory, a processor, and an emotion recognition program stored in the memory and executable on the processor, the emotion recognition program being configured to implement the emotion recognition method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores an emotion recognition program, which, when executed by a processor, implements the steps of the emotion recognition method as described in any one of claims 1 to 5.

Images