Emotion recognition method and device, equipment and storage medium

Abstract

The invention discloses an emotion recognition method and device, equipment and a storage medium. According to the method, image segmentation is carried out on a part of a face image of a user based on a preset image segmentation model, a complete face contour segmentation map of a current frame is obtained, and the part of the face image is an image formed by shielding a part of the face after the user wears the wearable device; constructing a space-time sequence based on the complete face contour segmentation map of the current frame and the complete face contour segmentation map of the historical frame; and performing emotion recognition on the space-time sequence based on a preset expression prediction model to obtain an emotion category of the user. Compared with a traditional emotion recognition method which cannot finish accurate recognition of the user emotion under the condition that the face is shielded, the method has the advantages that the full face expression in the equipment wearing scene is tracked through the pre-constructed image segmentation model and the expression prediction model, accurate emotion recognition is achieved, and the cross-user generalization ability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to an emotion recognition method and device, equipment and a storage medium. BACKGROUND

[0002] In virtual reality (VR), augmented reality (AR) and telecommunication, real-time and accurate facial expression tracking is essential for creating realistic virtual avatars. The existing technology faces two major challenges: occlusion problem: the frame or mask of the head-mounted device will occlude part of the user's face (such as forehead, eye), causing the traditional tracking algorithm based on the complete face to fail, and unable to complete the user emotion recognition. SUMMARY

[0003] The main purpose of the present application is to provide an emotion recognition method, device, equipment and storage medium, which aims to solve the technical problem that the traditional emotion recognition method cannot accurately recognize the user's emotion under the condition of facial occlusion.

[0004] To achieve the above purpose, the present application provides an emotion recognition method, which comprises the following steps: performing image segmentation on a partial face image of a user based on a preset image segmentation model to obtain a complete face contour segmentation map of a current frame, wherein the partial face image is an image formed by the user wearing a wearable device except for the occluded part of the face; constructing a spatio-temporal sequence based on the complete face contour segmentation map of the current frame and the complete face contour segmentation map of a historical frame; performing emotion recognition on the spatio-temporal sequence based on a preset expression prediction model to obtain an emotion category of the user.

[0005] Optionally, the emotion recognition based on the preset expression prediction model on the spatio-temporal sequence to obtain the emotion category of the user comprises: performing expression prediction on the spatio-temporal sequence based on a preset expression prediction model to obtain an expression prediction vector; determining an emotion difference vector based on the expression prediction vector and a universal reference neutral vector; performing emotion recognition according to the emotion difference vector and a preset Voronoi diagram to obtain an emotion category.

[0006] Optionally, the emotion recognition according to the emotion difference vector and the preset Voronoi diagram to obtain the emotion category comprises: calculating the Euclidean distance between the emotion difference vector and the difference center point corresponding to each emotion category in the preset Voronoi diagram; determine a nearest center point from the difference center points according to the Euclidean distance and a nearest principle; determine an emotion category according to an emotion label associated with a cell corresponding to the nearest center point.

[0007] Optionally, before the image segmentation of the partial face image of the user based on the preset image segmentation model to obtain the complete face contour segmentation graph of the current frame, the method comprises: capture a partial face image sample of a user wearing a mask based on a preset average frame rate; perform emotion recognition on the complete face image sample of the user wearing a mask to obtain a full-face facial feature parameter and an emotion label; construct a preset training set based on the partial face image, the full-face facial feature parameter and the emotion label; train an initial image segmentation model based on the preset training set to obtain a preset image segmentation model; train an initial expression prediction model based on the preset training set to obtain a preset expression prediction model; extract a full-face facial feature parameter with a neutral expression from the preset training set, and construct a preset Voronoi diagram based on the full-face facial feature parameter with the neutral expression.

[0008] Optionally, the training of the initial image segmentation model based on the preset training set to obtain the preset image segmentation model comprises: randomly select a preset proportion of partial face images from the preset training set, and obtain a labeled face contour segmentation graph; construct a preset segmentation training set according to the partial face images and the labeled face contour segmentation graph; train the initial image segmentation model based on the preset segmentation training set and a preset Dice loss function to obtain a preset image segmentation model, wherein the initial image segmentation model adopts ResNet18-FPN as a segmentation network architecture model.

[0009] Optionally, the training of the initial expression prediction model based on the preset training set to obtain the preset expression prediction model comprises: perform active frame labeling on the preset training set to obtain a target training set containing active frames and non-active frames; The initial expression prediction model is trained based on the target training set, preset weight factors, and preset weighted Huber loss function to obtain a preset expression prediction model. The initial prediction model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and then connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

[0010] Optionally, the step of extracting full-face facial feature parameters labeled as neutral expressions from the preset training set, and constructing a preset Voronoi diagram based on the full-face facial feature parameters of the neutral expressions, includes: Extract full-face facial feature parameters labeled as neutral expression from the preset training set; A general baseline neutral vector is calculated based on the full-face facial feature parameters of the neutral expression; Calculate the emotion difference vector corresponding to each emotion category based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general benchmark neutral vector. The center point of the difference for each emotion category is determined based on the mean of the emotion difference vectors corresponding to each emotion category. A spatial index for the target dimension is constructed based on the difference center points corresponding to each emotion category, and a Voronoi diagram cell is constructed based on the spatial index and the emotion labels corresponding to the difference center points; A preset Voronoi diagram is constructed based on the Voronoi diagram cells.

[0011] Furthermore, to achieve the above objectives, this application also provides an emotion recognition device, which includes: The image segmentation module is used to segment a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame. The portion of the facial image is the image formed after the user wears the wearable device, excluding the part of the face that is obscured. The sequence construction module is used to construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames. The emotion recognition module is used to perform emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category.

[0012] In addition, to achieve the above objectives, this application also proposes an emotion recognition device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the emotion recognition method described above.

[0013] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the emotion recognition method.

[0014] The one or more technical solutions provided by the application have at least the following technical effects: The application performs image segmentation on the partial face image of the user based on a preset image segmentation model to obtain a complete face contour segmentation graph of the current frame, wherein the partial face image is an image formed by shielding part of the face of the user after wearing the wearable device; a space-time sequence is constructed based on the complete face contour segmentation graph of the current frame and the complete face contour segmentation graph of the historical frame; and emotion recognition is performed on the space-time sequence based on a preset expression prediction model to obtain the emotion category of the user. Compared with the traditional emotion recognition method which cannot complete accurate recognition of the emotion of the user in the case of face shielding, the application realizes accurate emotion recognition by tracking the full-face expression in the device wearing scenario through the pre-constructed image segmentation model and expression prediction model, and improves the cross-user generalization capability. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0017] Figure 1 The flowchart provided for the first embodiment of the emotion recognition method of the application; Figure 2 The flowchart provided for the second embodiment of the emotion recognition method of the application; Figure 3 The flowchart provided for the third embodiment of the emotion recognition method of the application; Figure 4 The overall scheme flowchart provided for the third embodiment of the emotion recognition method of the application; Figure 5 The module structure diagram of the emotion recognition device of the embodiment of the application; Figure 6 The device structure diagram of the hardware running environment involved in the emotion recognition method in the embodiment of the application.

[0018] The objectives, functional features and advantages of the present application will be further illustrated in conjunction with the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are merely exemplary and not intended to limit the present application.

[0020] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments.

[0021] The main solution of the embodiments of the present application is: based on a preset image segmentation model, the present application performs image segmentation on part of the facial image of the user to obtain a complete facial contour segmentation graph of the current frame, wherein the part of the facial image is an image formed by shielding part of the face after the user wears the wearable device; based on the complete facial contour segmentation graph of the current frame and the complete facial contour segmentation graph of the historical frame, a space-time sequence is constructed; based on a preset expression prediction model, the space-time sequence is subjected to emotion recognition to obtain the emotion category of the user.

[0022] In the present embodiment, for the convenience of description, the following describes the computing service device as the execution subject.

[0023] Since the traditional emotion recognition method cannot complete the accurate recognition of the user's emotion in the case of facial occlusion.

[0024] The present application provides a solution, which tracks the full-face expression in the device wearing scene through the pre-constructed image segmentation model and expression prediction model, realizes accurate emotion recognition, and improves the cross-user generalization ability.

[0025] From the above embodiments, based on a preset image segmentation model, the present application performs image segmentation on part of the facial image of the user to obtain a complete facial contour segmentation graph of the current frame, wherein the part of the facial image is an image formed by shielding part of the face after the user wears the wearable device; based on the complete facial contour segmentation graph of the current frame and the complete facial contour segmentation graph of the historical frame, a space-time sequence is constructed; based on a preset expression prediction model, the space-time sequence is subjected to emotion recognition to obtain the emotion category of the user. Compared with the traditional emotion recognition method which cannot complete the accurate recognition of the user's emotion in the case of facial occlusion, the present application tracks the full-face expression in the device wearing scene through the pre-constructed image segmentation model and expression prediction model, realizes accurate emotion recognition, and improves the cross-user generalization ability.

[0026] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, or an electronic device capable of realizing the above functions, a device including an emotion recognition system, etc. The following will take a computer as an example to describe the embodiment and the following embodiments.

[0027] Based on this, the embodiment of the present application provides an emotion recognition method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the emotion recognition method of the present application is shown in the figure.

[0028] In the embodiment, the emotion recognition method includes steps S10-S30: Step S10, based on a preset image segmentation model, the partial face image of the user is segmented, and the complete face contour segmentation graph of the current frame is obtained.

[0029] It should be noted that the preset image segmentation model is a model trained based on ResNet18-FPN as a segmentation network architecture, which is used for segmentation and prediction processing of partial face images to obtain a complete face contour segmentation image. The model accurately extracts the face region and provides clean input for subsequent processing. The partial face image of the user wearing the wearable device is segmented and identified by the preset image segmentation model, and the complete face contour segmentation graph of the current frame is obtained. The wearable device can be a device for capturing the face image of the wearer, such as smart glasses, VR head-mounted display, smart earphones, etc. The complete face contour segmentation graph is a binary contour graph containing only the face skin region after removing all occlusions.

[0030] In a specific implementation, the real-time video stream is obtained by the corresponding camera of the wearable device, and each frame of the partial face image in the video stream is input into the preset image segmentation model for segmentation and prediction processing to obtain the complete face contour segmentation image. Therefore, the image segmentation model in the present application can predict the occluded part, and the model output is a binary segmentation graph, where the pixel value of 1 corresponds to the predicted face skin region, and the pixel value of 0 corresponds to all non-face regions including glasses. In this way, the system realizes the robustness of the glasses occlusion and provides a standardized face contour input without interference from glasses for subsequent continuous Blendshape (facial muscle action) prediction.

[0031] Step S20, based on the complete face contour segmentation graph of the current frame and the complete face contour segmentation graph of the historical frame, a space-time sequence is constructed.

[0032] It should be noted that, in order to improve the accuracy of expression prediction, the complete face contour segmentation graph of the current frame and the complete face contour segmentation graph of the historical frame are stacked to form a space-time sequence reflecting dynamic changes, and the constructed space-time sequence is input into a preset expression prediction model, so as to identify the facial expression. The complete face contour segmentation graph of the historical frame refers to the segmentation graph of the previous N-1 frames saved in the circular buffer.

[0033] In step S30, the emotion of the user is identified based on the preset expression prediction model, and the emotion category of the user is obtained.

[0034] It should be noted that the preset expression prediction model is a ResNet34_3D model obtained by improving the ResNet34 model in advance and training the improved model. The ResNet34_3D model is obtained by replacing all 2D convolution layers in the ResNet34 model structure with corresponding 3D convolution layers, replacing the batch normalization layer with a 3D batch normalization layer, and then connecting a flattening layer and a fully connected layer at the end of the 3D convolution feature extraction network.

[0035] It can be understood that the model is used to capture subtle and dynamic expression features, and infer and predict the expression features, and output an expression prediction vector corresponding to the current frame, wherein the expression prediction vector is a standardized 52-dimensional Blendshape facial action vector.

[0036] In a specific implementation, the space-time sequence is input into the expression prediction model, a 52-dimensional facial action vector corresponding to the current frame is output, and emotion recognition is performed based on the 52-dimensional Blendshape facial action vector corresponding to the current frame, to obtain the emotion category of the user. The finally output 52 Blendshape parameters can be directly used to drive compatible virtual images (such as characters in Unity / UnrealEngine). At the same time, the output emotion category and intensity information can be used to trigger the preset emotion animation of the virtual image, adjust the dialogue strategy of the interactive system, or record the emotional change trend of the user in the mental health application. All processing can be completed in real time on the edge device to meet the interactive needs of AR / VR applications.

[0037] In an implementable manner, the present application can also be quickly personalized for new users. When a new user uses the system, a personalization adaptation process is started, which includes a. Data collection: prompt the user to calibrate for about 1 minute, during which the user naturally makes a neutral expression and several basic expressions. The emotion recognition system synchronously collects part of the facial image. b. Model fine-tuning: using the LoRA technology, only add a trainable rank decomposition matrix to the last fully connected layer of the trained Blendshape expression prediction model (ResNet34_3D), and freeze all weights of the original model. The LoRA module is fine-tuned using the data collected from the new user, so that the predicted full-face expression is more accurate. The introduction of the fine-tuning mechanism and the personalized benchmark vector enables the system to quickly adapt to new users, while maintaining universality and providing more accurate expression tracking and emotion recognition experience for individuals.

[0038] The embodiment is based on a preset image segmentation model to perform image segmentation on the partial facial image of the user, to obtain a complete facial contour segmentation graph of the current frame, wherein the partial facial image is an image formed by shielding part of the face of the user after wearing the wearable device; construct a spatio-temporal sequence based on the complete facial contour segmentation graph of the current frame and the complete facial contour segmentation graph of the historical frame; perform emotion recognition on the spatio-temporal sequence based on a preset expression prediction model to obtain the emotion category of the user. Compared with the traditional emotion recognition method which cannot accurately recognize the emotion of the user under the condition of facial occlusion, the embodiment tracks the full-face expression under the device wearing scene through the pre-constructed image segmentation model and expression prediction model, realizes accurate emotion recognition, effectively solves the tracking problem caused by facial occlusion under the device, and improves the cross-user robustness of emotion recognition through decoupling representation, while supporting efficient personalized fine-tuning, and is suitable for real-time virtual avatar driving, human-computer emotional interaction and other scenes.

[0039] Based on the above Figure 1 The first embodiment is shown, and the second embodiment of the emotion recognition method of the present application is proposed; refer to Figure 2 , Figure 2 The flowchart of the second embodiment of the emotion recognition method of the present application. Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above first embodiment can be referred to the above introduction, and will not be described in detail hereinafter.

[0040] In the embodiment, the step S30 further includes: Step S301, performing expression prediction on the spatio-temporal sequence based on a preset expression prediction model to obtain an expression prediction vector.

[0041] It should be noted that the trained ResNet34_3D model is used to infer and predict facial features, and outputs the facial prediction vector corresponding to the current frame. The facial prediction vector is a standardized 52-dimensional Blendshape facial motion vector B_pred.

[0042] Step S302: Determine the emotion difference vector based on the expression prediction vector and the general benchmark neutral vector.

[0043] It should be noted that the universal baseline neutral vector can be a pre-calculated neutral vector B_neutral based on the full-face facial feature parameters of a neutral expression. The emotion difference vector is determined by calculating the difference between the expression prediction vector and the universal baseline neutral vector.

[0044] Understandably, the formula for calculating the emotion difference vector of the current facial expression is as follows: δ_pred = B_pred - B_neutral.

[0045] Step S303: Perform emotion recognition based on the emotion difference vector and the preset Voronoi diagram to obtain the emotion category.

[0046] Understandably, the calculated emotion difference vector of the current expression is mapped onto a preset Voronoi diagram, and emotion recognition is performed based on the emotion difference vector and each emotion center point in the preset Voronoi diagram to obtain the emotion category.

[0047] It should be understood that the preset Voronoi diagram is a pre-set Voronoi diagram used to realize emotion categories. This Voronoi diagram is a spatial index structure for the target dimension built based on the full-face facial features of neutral expressions and the full-face facial feature parameters corresponding to all emotions. A general baseline neutral vector is calculated through the full-face facial feature parameters of neutral expressions. Based on the general baseline neutral vector and the full-face facial feature parameters corresponding to all emotions, the emotion difference vector corresponding to each emotion category is calculated. The difference center point corresponding to each emotion category is determined based on the emotion difference vector. Thus, a spatial index structure for the target dimension is constructed based on the difference center point. The Voronoi diagram cells are constructed based on the spatial index result and the difference center point to generate the Voronoi diagram.

[0048] Furthermore, step S303 further includes: calculating the Euclidean distance between the emotion difference vector and the difference center points corresponding to each emotion category in the preset Voronoi diagram; determining the nearest center point from the difference center points according to the Euclidean distance and the nearest distance principle; and determining the emotion category according to the emotion label associated with the cell corresponding to the nearest center point.

[0049] It should be noted that the Voronoi diagram transforms the complex classification problem into a nearest neighbor query in a high-dimensional space. The decision-making process is transparent, and the cells can intuitively display the distribution relationship of different emotions in the Blendshape difference space. In the process of querying and classifying emotions based on the Voronoi diagram, the emotion difference vector δ_pred is mapped to a preloaded Voronoi diagram. The Euclidean distance between the emotion difference vector δ_pred and the difference center point {C_emotion} corresponding to each emotion category is calculated, and the nearest center point is selected from among all center points based on the principle of proximity. The emotion difference vector δ_pred is classified into the Voronoi cell corresponding to this center point, and the emotion label associated with this cell is the dominant emotion identified by the system. In specific implementation, this application can also achieve emotion intensity estimation by simultaneously calculating the L2 norm (modulus) of δ_pred as a primary indicator of emotion intensity. The larger the norm, the greater the amplitude of facial expression changes, and the stronger the emotion may be. More precisely, the relative distance from δ_pred to the center of its assigned cell can be calculated.

[0050] This embodiment segments a portion of a user's facial image using a pre-defined image segmentation model to obtain a complete facial contour segmentation map of the current frame. The partial facial image is formed when the user's face is partially obscured by wearing a wearable device. A spatiotemporal sequence is constructed based on the complete facial contour segmentation map of the current frame and complete facial contour segmentation maps of historical frames. An expression prediction vector is obtained by predicting expressions based on the spatiotemporal sequence using a pre-defined expression prediction model. An emotion difference vector is determined based on the expression prediction vector and a general baseline neutral vector. Emotion recognition is then performed based on the emotion difference vector and a pre-defined Voronoi diagram to obtain the emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this embodiment tracks full-face expressions in device-wearing scenarios using a pre-constructed image segmentation model and expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities.

[0051] Based on the above Figure 1 The first embodiment shown illustrates a third embodiment of the emotion recognition method of this application; see also... Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the emotion recognition method of this application. Based on the first embodiment of this application, in the third embodiment, the content that is the same as or similar to the first embodiment described above can be referred to the above description and will not be repeated hereafter.

[0052] In this embodiment, the method further includes the following steps before step S10: Step S01: Capture partial facial image samples of the wearer based on a preset average frame rate.

[0053] It should be noted that during the construction of the training set, dataset collection can be completed based on a pre-built data acquisition system. This system can include three hardware components: a smart wearable device, such as smart glasses (containing dual low-power cameras), a smart terminal, such as a smartphone (running a face capture application like Live Link Face), and a PC as a control center. The preset average frame rate is a pre-set statistical average of the number of image frames captured per second by the smart wearable device's cameras over a period of time. This frame rate can be adjusted according to the performance of the smart wearable device.

[0054] In practice, the data acquisition process is uniformly scheduled through a PC-based control program. When acquisition begins, the glasses' camera captures facial images of the wearer's face, excluding the obscured areas, at a preset average frame rate (e.g., 30fps) as partial facial image samples.

[0055] Step S02: Perform emotion recognition on the complete facial image sample of the wearer to obtain full-face facial feature parameters and emotion labels.

[0056] Understandably, while capturing partial facial images of the wearer through the smart wearable device's camera, the smart terminal's camera simultaneously captures the wearer's complete facial image, obtaining a complete facial image sample. The corresponding expression recognition application on the smart terminal then performs emotion recognition on the complete facial image sample, obtaining full-face facial feature parameters and emotion labels. Data alignment is achieved through timestamps and network protocols. The complete facial image refers to the facial image of the user while wearing the smart wearable device.

[0057] While capturing partial facial images of the wearer, the system simultaneously controls the smartphone's front-facing camera to take a full frontal shot of the same user. The system then uses the smartphone's built-in Live Link Face application to analyze and output 52 Blendshape parameters as real-world facial expression data in real time. Each frame is labeled with a corresponding emotion tag. These emotion tags can include joy, sadness, anger, surprise, disgust, fear, and neutrality.

[0058] Step S03: Construct a preset training set based on the partial facial images, the full-face facial feature parameters, and the emotion labels.

[0059] It should be noted that a preset training set is constructed by acquiring partial facial images, the full-face facial feature parameters, and emotion labels.

[0060] Step S04: Train the initial image segmentation model based on the preset training set to obtain the preset image segmentation model.

[0061] It should be noted that the initial image segmentation model uses ResNet18-FPN as the segmentation network architecture. The initial image segmentation model is trained using a pre-defined training set to obtain the pre-defined image segmentation model.

[0062] Furthermore, to illustrate the segmentation model training process, step S04 further includes: randomly selecting a portion of facial images of a preset proportion from the preset training set, and obtaining an annotated facial contour segmentation map; constructing a preset segmentation training set based on the portion of facial images and the annotated facial contour segmentation map; training an initial image segmentation model based on the preset segmentation training set and a preset Dice loss function to obtain a preset image segmentation model, wherein the initial image segmentation model uses ResNet18-FPN as the segmentation network architecture.

[0063] It should be noted that a portion of facial images of a preset proportion are randomly selected from the preset training set. This preset proportion can be set according to training needs, such as 5%. The preset Dice loss function is a pre-set metric used for fitting and statistically analyzing the pixel set. This application determines the number of pixels in the overlapping part of the predicted and real faces, as well as the sum of the total number of pixels in the predicted and real faces, by fitting the predicted and real face pixel set output by the model. The Dice coefficient is obtained by dividing twice the number of pixels in the overlapping part of the predicted and real faces by the sum of the total number of pixels in the predicted and real faces. The formula for calculating the Dice coefficient is as follows: ; Where P: the set of pixels predicted by the model as a face. G: the set of pixels of the real face. |P∩G|: the number of pixels overlapping between the predicted and real faces (intersection). |P| + |G|: the sum of the total number of pixels of the predicted face and the real face. The Dice coefficient ranges from [0,1], where 1 represents a perfect match, completely overlapping. 0 indicates no overlap at all. Since a larger Dice coefficient is better, while a smaller loss function is better, the Dice loss is usually defined as: L Dice =1 Dice(P,G) is a minimizeable Dice loss function obtained by maximizing the overlap between the prediction and the true label.

[0064] Understandably, approximately 5% of facial images are randomly selected from a pre-defined training set, and facial contour segmentation maps are manually labeled to construct a segmentation training set for training a model that can predict complete facial contours from partial images. ResNet18-FPN is used as the segmentation network architecture, with a 320×240 pixel partial facial image as input and a binary segmentation map of the same size as output (face region = 1, background = 0). The Dice loss function is used for training until the model can accurately predict complete facial contours from occluded images.

[0065] Step S05: Train the initial expression prediction model based on the preset training set to obtain the preset expression prediction model.

[0066] It should be noted that the initial expression prediction model is trained using a preset training set to obtain the preset expression prediction model. The initial expression model is an improved version of the ResNet34 model, and the preset expression prediction model is a model trained using the improved ResNet34 model.

[0067] Furthermore, to illustrate the training process of the preset expression prediction model, step S05 further includes: labeling the preset training set with active frames to obtain a target training set containing active and inactive frames; training the initial expression prediction model based on the target training set, preset weight factors, and preset weighted Huber loss function to obtain the preset expression prediction model. The initial prediction model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and sequentially connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

[0068] It should be noted that the expression prediction model is trained using a preset training set. This is achieved by inputting the complete training set into the trained segmentation model and outputting a predicted segmentation map. The segmentation image of the current frame is stacked with the segmentation images of the previous N frames in chronological order to form a spatiotemporal sequence of the target size. This spatiotemporal sequence is then input into the initial expression prediction model for training. During training, a weighted Huber loss function is used to balance the regression difficulty between active expression frames and neutral frames.

[0069] Understandably, for each sample in the pre-defined training set, its emotion label is used to determine whether it is an active frame. Samples with a "neutral" emotion label are labeled as inactive frames, while samples with other emotion labels (such as happy, sad, etc.) are labeled as active frames. The samples labeled with active / inactive frames are then integrated with the original data (including partial facial images, complete facial images, ground truth values ​​of Blendshape parameters, and emotion labels) to form the target training set. Based on the target training set and the pre-defined weighted Huber loss function, the initial expression prediction model is trained to obtain the pre-defined expression prediction model.

[0070] It should be understood that, by pre-setting weight factors, two weight factors are calculated based on the proportion of active and inactive frames in the target training set: f n (Active frame weights) and f a (Inactive frame weight). Specific calculation method: Let the number of active frames be N. n The number of inactive frames is N a If the number of emotion categories is K (excluding neutral), then ; These weights will be used to weight the Huber loss function to balance the contributions of active and inactive frames to the loss function. The Huber loss function combines the mean error (MSE) and the mean absolute error (MAE), and its formula is as follows: ; Here, δ is a hyperparameter that controls the switching point from MSE to MAE.

[0071] The pre-defined weighted Huber loss function is: ; Where B is the batch size, i n (y) is an indicator function, which is 1 when sample i is an inactive frame. y is the true value, where i is the actual value. a Similarly, f(x) is the model's predicted value. These are the weights of the facial expressions in the active frames, where K is the number of facial expression types in the dataset. It is the number of active frames. It is the number of inactive frames. This indicates that the current frame is the active frame. The weights are for inactive frames, i.e., frames with natural facial expressions. This indicates that the current frame is inactive. 0) is a hyperparameter used to control the switching point of Huber loss between MSE and MAE. It occurs when the absolute value of the error between the predicted and actual values ​​is less than or equal to... When the absolute value of the error is greater than 1, use mean squared error; when the absolute value of the error is greater than 1. When using linear functions, the impact of outliers on the loss can be reduced.

[0072] The ResNet34_3D model is initialized by replacing 2D convolutional layers with 3D convolutional layers and batch normalization layers with 3D batch normalization layers. A flattening layer and a fully connected layer are then added to the network, outputting a 52-dimensional vector. Data from the target training set is organized into batches, each batch containing B samples. Each sample is a segmentation map of N consecutive frames (obtained from the first-stage segmentation model) and the corresponding ground truth 52-dimensional Blendshape parameters. For each batch: a. Forward propagation: The N-frame segmentation map sequence is input into the model to obtain a 52-dimensional Blendshape prediction vector. b. Loss calculation: The weighted Huber loss is calculated based on the ground truth Blendshape parameters and the predicted values. c. Backpropagation: The gradient is calculated and the model parameters are updated. This process is repeated for multiple training epochs until the model converges, obtaining the preset facial expression prediction model.

[0073] In the specific implementation, the glasses image is input into a pre-trained segmentation model to obtain a predicted segmentation map. This segmentation map is then stacked chronologically with the segmentation maps of the preceding N frames (e.g., N=4) to form a spatiotemporal input tensor of size N×320×240. A 3D convolutional network based on ResNet34 (ResNet34_3D) is constructed: all 2D convolutional layers in the original ResNet34 are replaced with corresponding 3D convolutional layers, and batch normalization layers are replaced with 3D batch normalization layers. The network is finally connected to flattening layers and fully connected layers, outputting a 52-dimensional vector. When the spatiotemporal sequence is input into the ResNet34_3D network, the network outputs a 52-dimensional blendshape predicted value.

[0074] Step S06: Extract full-face facial feature parameters labeled as neutral expression from the preset training set, and construct a preset Voronoi diagram based on the full-face facial feature parameters of the neutral expression.

[0075] It should be noted that in the Voronoi emotion difference space construction stage of this application, after the model training is completed, a Voronoi space for emotion recognition is constructed by extracting full-face facial feature parameters with neutral emotion labels from the preset training set, and constructing a preset Voronoi map based on the full-face facial feature parameters of neutral emotions.

[0076] In the specific implementation, to further illustrate the training construction and online inference process after the construction is completed, refer to... Figure 4The schematic diagram of the overall solution shows that data acquisition and synchronous acquisition of partially occluded facial images captured by a wearable device, as well as ground truth values ​​of full-face blendshape parameters and emotion labels generated by an independent facial capture system, are used to train a two-stage neural network. The first stage uses a ResNet18-FPN network to segment facial contours based on partial facial images. The second stage uses a ResNet34 model network to replace the 3D convolutional layers and predicts the 52-dimensional blendshape parameters of the full face based on facial contours from multiple consecutive frames. Based on the ground truth values ​​of blendshape in the training data, a general baseline neutral blendshape vector is calculated. The difference between the blendshape vector of each emotion sample and the baseline vector is calculated. A Voronoi diagram centered on the emotion category is constructed based on the difference vector. For the input partial facial images, the two-stage network is used sequentially to obtain the predicted blendshape vector. The difference between the predicted vector and the baseline vector is calculated. The difference vector is mapped to the Voronoi diagram, and the emotion category is determined based on the Voronoi cell it falls into. A general baseline neutral vector is calculated based on the full-face facial feature parameters of neutral expressions. For each emotion category in the training set, the mean of all corresponding difference vectors is calculated based on the general baseline neutral vector to obtain the center point of that emotion category. A spatial index structure is then constructed based on this center point. This spatial index structure and the emotion labels corresponding to each center point are serialized and stored as a data file, thus obtaining the preset Voronoi diagram.

[0077] Further, step S06 also includes: extracting full-face facial feature parameters of neutral expression from the preset training set; calculating a general baseline neutral vector based on the full-face facial feature parameters of the neutral expression; calculating the emotion difference vector corresponding to each emotion category based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general baseline neutral vector; determining the difference center point corresponding to each emotion category based on the mean of the emotion difference vectors corresponding to each emotion category; constructing a spatial index of the target dimension based on the difference center points corresponding to each emotion category, and constructing Voronoi diagram cells based on the spatial index and the emotion labels corresponding to the difference center points; and constructing a preset Voronoi diagram based on the Voronoi diagram cells.

[0078] It should be noted that, from the preset training set, full-face facial feature parameters with neutral emotion labels are extracted to calculate a general baseline neutral vector. Based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general baseline neutral vector, the emotion difference vector corresponding to each emotion category is calculated. The mean of the emotion difference vectors corresponding to each emotion category is used to determine the difference center point corresponding to each emotion category. The difference center point corresponding to each emotion category is used as the generation point to construct a spatial index for the target dimension. Based on the spatial index and the emotion label corresponding to the difference center point, a Voronoi diagram cell is constructed. Based on the Voronoi diagram cell, a preset Voronoi diagram is constructed.

[0079] In the specific implementation, the Voronoi space construction process is as follows: a. Calculate the general baseline neutral vector: Extract the true Blendshape values ​​corresponding to all samples labeled as "neutral" expressions from the training dataset. Calculate the arithmetic mean of these vectors to obtain a 52-dimensional general baseline neutral Blendshape vector, denoted as B_neutral. b. Calculate the emotion difference vector: For each sample with an emotion label in the dataset, obtain its Blendshape vector B_i, and calculate its emotion difference vector δ_i = B_i - B_neutral. This operation aims to strip away individual static facial features and obtain the amount of expression change purely driven by emotion. c. Calculate the emotion category centroid: Group all samples according to their emotion labels. For each basic emotion category (such as happiness, sadness, anger, surprise, etc.), calculate the mean of the difference vector δ_i of all samples belonging to that category to obtain the difference centroid C_emotion for that emotion. d. Constructing the Voronoi diagram: The difference center points {C_happy, C_sad, C_angry, ...} across all emotion categories are used as seed points. In the 52-dimensional difference vector space, the Voronoi partitioning algorithm is applied: any point in the space belongs to the region represented by its nearest seed point. This generates a Voronoi diagram, where each convex polygon region (Voronoi cell) corresponds to an emotion category. This diagram structure is serialized and saved for subsequent inference.

[0080] This embodiment captures partial facial image samples of the user based on a preset average frame rate; performs emotion recognition on the complete facial image samples of the user to obtain full-face facial feature parameters and emotion labels; constructs a preset training set based on partial facial images, full-face facial feature parameters, and emotion labels; trains an initial image segmentation model based on the preset training set to obtain a preset image segmentation model; trains an initial expression prediction model based on the preset training set to obtain a preset expression prediction model; extracts full-face facial feature parameters with neutral emotion labels from the preset training set, and constructs a preset Voronoi diagram based on the full-face facial feature parameters of neutral expressions; performs image segmentation on partial facial images of the user based on the preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, wherein partial facial images are images formed when the user's face is partially obscured after wearing the wearable device; constructs a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames; performs emotion recognition on the spatiotemporal sequence based on the preset expression prediction model to obtain the user's emotion category. Compared to traditional emotion recognition methods that cannot accurately identify user emotions when the face is occluded, this embodiment tracks full-face expressions in device-wearing scenarios using a pre-built image segmentation model and expression prediction model, achieving accurate emotion recognition and improving cross-user generalization capabilities.

[0081] This application also provides an emotion recognition device; please refer to [reference needed]. Figure 5 The emotion recognition device includes: Image segmentation module 10 is used to segment a portion of a user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, wherein the portion of the facial image is an image formed after the user wears a wearable device, excluding the part of the face that is obscured. Sequence construction module 20 is used to construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation map of the historical frames; The emotion recognition module 30 is used to perform emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category.

[0082] Furthermore, the emotion recognition module 30 is also used to predict the expression of the spatiotemporal sequence based on a preset expression prediction model to obtain an expression prediction vector; determine an emotion difference vector based on the expression prediction vector and a general benchmark neutral vector; and perform emotion recognition based on the emotion difference vector and a preset Voronoi diagram to obtain an emotion category.

[0083] Furthermore, the emotion recognition module 30 is also used to calculate the Euclidean distance between the emotion difference vector and the difference center point corresponding to each emotion category in the preset Voronoi diagram; determine the nearest center point from the difference center points according to the Euclidean distance and the nearest distance principle; and determine the emotion category according to the emotion label associated with the cell corresponding to the nearest center point.

[0084] Furthermore, the emotion recognition device also includes a model building module, which is used to capture partial facial image samples of the wearer based on a preset average frame rate; perform emotion recognition on the complete facial image samples of the wearer to obtain full-face facial feature parameters and emotion labels; construct a preset training set based on the partial facial images, the full-face facial feature parameters, and the emotion labels; train an initial image segmentation model based on the preset training set to obtain a preset image segmentation model; train an initial expression prediction model based on the preset training set to obtain a preset expression prediction model; extract full-face facial feature parameters with neutral emotion labels from the preset training set, and construct a preset Voronoi diagram based on the full-face facial feature parameters of neutral expressions.

[0085] Furthermore, the model building module is also used to randomly select a portion of face images of a preset proportion from the preset training set and obtain an annotated facial contour segmentation map; construct a preset segmentation training set based on the portion of face images and the annotated facial contour segmentation map; train the initial image segmentation model based on the preset segmentation training set and the preset Dice loss function to obtain a preset image segmentation model, wherein the initial image segmentation model adopts ResNet18-FPN as the segmentation network architecture.

[0086] Furthermore, the model building module is also used to annotate the preset training set with active frames to obtain a target training set containing active and inactive frames; and to train the initial expression prediction model based on the target training set, preset weight factors, and preset weighted Huber loss function to obtain a preset expression prediction model. The initial prediction model is obtained by replacing all 2D convolutional layers in the ResNet34 model structure with corresponding 3D convolutional layers, replacing the batch normalization layer with a 3D batch normalization layer, and then sequentially connecting a flattening layer and a fully connected layer at the end of the 3D convolutional feature extraction network.

[0087] Furthermore, the model building module is also used to extract full-face facial feature parameters with neutral emotion labels from the preset training set; calculate a general baseline neutral vector based on the full-face facial feature parameters of the neutral expression; calculate the emotion difference vector corresponding to each emotion category based on the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general baseline neutral vector; determine the difference center point corresponding to each emotion category based on the mean of the emotion difference vectors corresponding to each emotion category; construct a spatial index of the target dimension based on the difference center points corresponding to each emotion category, and construct Voronoi diagram cells based on the spatial index and the emotion labels corresponding to the difference center points; and construct a preset Voronoi diagram based on the Voronoi diagram cells.

[0088] The emotion recognition device provided in this application, employing the emotion recognition method described in the above embodiments, can solve the technical problem that traditional emotion recognition methods cannot accurately identify user emotions when the face is obscured. Compared with the prior art, the beneficial effects of the emotion recognition device provided in this application are the same as those of the emotion recognition method provided in the above embodiments, and other technical features in the emotion recognition device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0089] This application provides an emotion recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the emotion recognition method in the first embodiment described above.

[0090] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the emotion recognition device in the embodiments of this application. The emotion recognition device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The emotion recognition device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0091] like Figure 6As shown, the emotion recognition device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the emotion recognition device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the emotion recognition device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows emotion recognition devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0092] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0093] The emotion recognition device provided in this application, employing the emotion recognition method described in the above embodiments, can solve the technical problem that traditional emotion recognition methods cannot accurately identify user emotions when the face is obscured. Compared with the prior art, the beneficial effects of the emotion recognition device provided in this application are the same as those of the emotion recognition method provided in the above embodiments, and other technical features of this emotion recognition device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0094] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0095] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0096] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the emotion recognition method in the above embodiments.

[0097] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0098] The aforementioned computer-readable storage medium may be included in the emotion recognition device; or it may exist independently and not incorporated into the emotion recognition device.

[0099] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the emotion recognition device, cause the emotion recognition device to: perform image segmentation on a portion of the user's facial image based on a preset image segmentation model to obtain a complete facial contour segmentation map of the current frame, wherein the portion of the facial image is an image formed after the user wears a wearable device, excluding the part of the face that is obscured; construct a spatiotemporal sequence based on the complete facial contour segmentation map of the current frame and the complete facial contour segmentation maps of historical frames; and perform emotion recognition on the spatiotemporal sequence based on a preset expression prediction model to obtain the user's emotion category.

[0100] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0102] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0103] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described emotion recognition method, thereby solving the technical problem of emotion recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the emotion recognition method provided in the above embodiments, and will not be repeated here.

[0104] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method of emotion recognition, characterized by, The method comprises: performing image segmentation on a partial face image of a user based on a preset image segmentation model to obtain a complete face contour segmentation graph of a current frame, wherein the partial face image is an image formed by a face occlusion part after the user wears a wearable device; constructing a space-time sequence based on the complete face contour segmentation graph of the current frame and complete face contour segmentation graphs of historical frames; performing emotion recognition on the space-time sequence based on a preset expression prediction model to obtain an emotion category of the user. 2.The emotion recognition method of claim 1, wherein, The emotion recognition on the space-time sequence based on the preset expression prediction model to obtain the emotion category of the user comprises: performing expression prediction on the space-time sequence based on the preset expression prediction model to obtain an expression prediction vector; determining an emotion difference vector based on the expression prediction vector and a general reference neutral vector; performing emotion recognition based on the emotion difference vector and a preset Voronoi diagram to obtain an emotion category. 3.The emotion recognition method of claim 2, wherein, The emotion recognition based on the emotion difference vector and the preset Voronoi diagram to obtain the emotion category comprises: calculating the Euclidean distance between the emotion difference vector and a difference center point corresponding to each emotion category in the preset Voronoi diagram; determining a nearest center point from the difference center points according to the Euclidean distance and a nearest distance principle; determining an emotion category according to an emotion label associated with a cell corresponding to the nearest center point. 4.The emotion recognition method of claim 1, wherein, Before the performing image segmentation on the partial face image of the user based on the preset image segmentation model to obtain the complete face contour segmentation graph of the current frame, the method comprises: capturing partial face image samples of a user wearing a wearable device based on a preset average frame rate; performing emotion recognition on complete face image samples of the user wearing the wearable device to obtain full-face facial feature parameters and emotion labels; constructing a preset training set based on the partial face images, the full-face facial feature parameters, and the emotion labels; training an initial image segmentation model based on the preset training set to obtain a preset image segmentation model; training an initial expression prediction model based on the preset training set to obtain a preset expression prediction model; extracting full-face facial feature parameters with a neutral expression from the preset training set, and constructing a preset Voronoi diagram based on the full-face facial feature parameters with the neutral expression. 5.The emotion recognition method of claim 4, wherein, The training of the initial image segmentation model based on the preset training set to obtain the preset image segmentation model comprises: randomly selecting a preset proportion of partial face images from the preset training set, and obtaining labeled face contour segmentation graphs; constructing a preset segmentation training set according to the partial face images and the labeled face contour segmentation graphs; training an initial image segmentation model based on the preset segmentation training set and a preset Dice loss function to obtain a preset image segmentation model, wherein the initial image segmentation model uses ResNet18-FPN as a segmentation network architecture model. 6.The emotion recognition method of claim 4, wherein, The training of the initial expression prediction model based on the preset training set to obtain the preset expression prediction model comprises: performing activity frame labeling on the preset training set to obtain a target training set containing activity frames and non-activity frames; Train the initial expression prediction model based on the target training set, a preset weight factor, and a preset weighted Huber loss function to obtain a preset expression prediction model. The initial prediction model is a model obtained by replacing all 2D convolution layers in a ResNet34 model structure with corresponding 3D convolution layers, replacing batch normalization layers with 3D batch normalization layers, and then connecting a flattening layer and a fully connected layer in sequence at the end of the 3D convolution feature extraction network. 7.The emotion recognition method of claim 4, wherein, The method further includes: extracting, from the preset training set, full-face facial feature parameters of a neutral expression with a neutral emotion label; and constructing a preset Voronoi diagram based on the full-face facial feature parameters of the neutral expression, including: extracting, from the preset training set, full-face facial feature parameters of a neutral expression with a neutral emotion label; calculating a general reference neutral vector according to the full-face facial feature parameters of the neutral expression; calculating an emotion difference vector corresponding to each emotion category according to the full-face facial feature parameters corresponding to all emotion labels in the preset training set and the general reference neutral vector; determining a difference center point corresponding to each emotion category according to a mean value of the emotion difference vector corresponding to each emotion category; constructing a spatial index of the target dimension based on the difference center points corresponding to each emotion category, and constructing a Voronoi diagram cell according to the spatial index and the emotion labels corresponding to the difference center points; constructing a preset Voronoi diagram based on the Voronoi diagram cell.

8. An emotion recognition apparatus characterized by comprising: The device includes: an image segmentation module configured to perform image segmentation on a partial facial image of a user based on a preset image segmentation model to obtain a complete facial contour segmentation map of a current frame, wherein the partial facial image is an image formed by the user wearing a wearable device except for a facial occlusion part; a sequence construction module configured to construct a spatio-temporal sequence based on the complete facial contour segmentation map of the current frame and complete facial contour segmentation maps of historical frames; an emotion recognition module configured to perform emotion recognition on the spatio-temporal sequence based on a preset expression prediction model to obtain an emotion category of the user.

9. An emotion recognition device, characterized by, The emotion recognition device includes a memory, a processor, and an emotion recognition program stored on the memory and executable on the processor, wherein the emotion recognition program is configured to implement the steps of the emotion recognition method according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium stores an emotion recognition program, and the emotion recognition program, when executed by a processor, implements the steps of the emotion recognition method according to any one of claims 1 to 7.

Images