Facial expression recognition model training method, employee evaluation method and system
By removing facial identity feature information during the training stage of the facial expression recognition model, the problem of identity information interference is solved and the accuracy of facial expression recognition of the model is improved.
Patent Information
- Application Number
- CN202411973626.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-29
AI Technical Summary
When the facial expression recognition model in the prior art extracts facial expression feature information, the identity information of the face will interfere with the accuracy and reduce the recognition accuracy of the model.
By extracting feature of the target frame of the sample video data and the facial slice image of the support frame, the facial identity feature information is removed, pure facial expression feature information is obtained, and the model weight parameters are updated through the loss function.
It effectively avoids the interference of identity information on facial expression feature information and improves the accuracy of facial expression recognition by the model.
Smart Images
Figure CN119904900A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of biometrics, and in particular to a facial expression recognition model training method, an employee evaluation method and a system. Background Art
[0002] In the service industry, the quality of service provided by employees directly affects customer satisfaction and brand image. With the continuous advancement of computer vision technology, the application of this technology in the field of employee training has gradually become possible. By analyzing employees' facial expressions, body language, and behavioral patterns, employees' service skills can be more effectively evaluated and improved.
[0003] In the prior art, it is a common practice in this field to identify facial expression feature information of a person's face through a facial expression recognition model, and then evaluate the employee's facial expression accordingly based on the facial expression feature information. Therefore, whether accurate facial expression feature information can be extracted directly affects the evaluation effect of facial expression.
[0004] However, in addition to facial expression feature information, a face also includes identity information of the face, such as certain geometric features of the face, differences in facial structure of different individuals, etc. In facial expression recognition, the identity information of the face may interfere with the accurate recognition of facial expression feature information, and the facial expression recognition model in the prior art usually directly extracts facial expressions from video frames. In this process, the identity information of the face will interfere with the facial expression feature information, reducing the accuracy of the facial expression recognition model in recognizing facial expressions. Summary of the invention
[0005] The present application provides a facial expression recognition model training method, an employee evaluation method and a system to solve the technical problem that the facial expression recognition model in the prior art usually directly extracts facial expressions from video frames, and in this process the identity information of the face interferes with the facial expression feature information, thereby reducing the accuracy of the facial expression recognition model in recognizing facial expressions.
[0006] In a first aspect, the present application provides a facial expression recognition model training method, the method comprising:
[0007] Performing a feature extraction operation on a facial slice image of a target frame of sample video data to obtain first pure facial expression feature information, and performing the feature extraction operation on a facial slice image of a support frame of the sample video data to obtain second pure facial expression feature information, wherein the feature extraction operation includes: converting the facial slice image into a feature map, extracting facial feature information and facial identity feature information in the feature map according to the feature map, removing the facial identity feature information from the facial feature information, and obtaining pure facial expression feature information;
[0008] According to the first pure facial expression feature information and the second pure facial expression feature information, a loss value is calculated by a loss function, and a weight parameter of the facial expression recognition model is updated according to the loss value.
[0009] In a second aspect, the present application provides an employee evaluation method, which is implemented based on a facial expression recognition model, and the facial expression recognition model is trained by the method described in the first aspect, and the employee evaluation method includes:
[0010] Extracting first facial expression feature information of the candidate employee video data and second facial expression feature information of the first sample video data through the facial expression recognition model, and calculating expression similarity scores of the first facial expression feature information and the second facial expression feature information through the facial expression recognition model;
[0011] Extracting first limb movement feature information of the candidate employee video data and second limb movement feature information of the second sample video data, and calculating a limb movement similarity score between the first limb movement feature information and the second limb movement feature information;
[0012] An evaluation result of the candidate employee is generated according to the facial expression similarity score and the body movement similarity score.
[0013] In a third aspect, the present application provides an employee evaluation system, the system is implemented based on a facial expression recognition model, the facial expression recognition model is trained by the method according to any one of claims 1 to 6, and the system comprises:
[0014] A facial expression recognition module, used to: extract first facial expression feature information of the candidate employee video data and second facial expression feature information of the first sample video data through the facial expression recognition model, and calculate the expression similarity score of the first facial expression feature information and the second facial expression feature information through the facial expression recognition model;
[0015] A body movement analysis module is used to: extract first body movement feature information of the candidate employee video data and second body movement feature information of the second sample video data, and calculate a body movement similarity score between the first body movement feature information and the second body movement feature information;
[0016] An evaluation module is used to generate an evaluation result of the candidate employee according to the expression similarity score and the body movement similarity score.
[0017] In a fourth aspect, the present application provides a facial expression recognition model training device, the device comprising:
[0018] A feature extraction module is used to: perform a feature extraction operation on a facial slice image of a target frame of sample video data to obtain first pure facial expression feature information, and perform the feature extraction operation on a facial slice image of a support frame of the sample video data to obtain second pure facial expression feature information, wherein the feature extraction operation includes: converting the facial slice image into a feature map, extracting facial feature information and facial identity feature information in the feature map according to the feature map, removing the facial identity feature information from the facial feature information, and obtaining pure facial expression feature information;
[0019] A weight updating module is used to calculate a loss value through a loss function according to the first pure facial expression feature information and the second pure facial expression feature information, and update a weight parameter of the facial expression recognition model according to the loss value.
[0020] In a fifth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;
[0021] The memory stores computer-executable instructions;
[0022] The processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect or the method described in the second aspect.
[0023] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect, or the method described in the second aspect.
[0024] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in the first aspect, or the method described in the second aspect.
[0025] The facial expression recognition model training method, employee evaluation method and system provided in the present application can avoid the interference of identity information on facial expression feature information by removing identity information from facial feature information during the training stage, thereby improving the model's focus on the dynamic changes of facial expressions and improving the model's accuracy in recognizing facial expressions. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0027] Figure 1A flowchart of a facial expression recognition model training method provided in an embodiment of the present application;
[0028] Figure 2 A schematic diagram of the facial expression recognition model structure provided in the embodiment of the present application;
[0029] Figure 3 A schematic diagram of a feature extraction operation method provided in an embodiment of the present application;
[0030] Figure 4 A schematic diagram of a flow chart of an employee evaluation method provided in an embodiment of the present application;
[0031] Figure 5 A schematic diagram of a body movement recognition model structure provided in an embodiment of the present application;
[0032] Figure 6 A schematic diagram of the structure and workflow of an employee evaluation system provided in an embodiment of the present application;
[0033] Figure 7 A schematic diagram of the structure of a facial expression recognition model training device provided in an embodiment of the present application;
[0034] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.
[0035] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0036] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0037] It should be noted that the present application provides a facial expression recognition model training method, an employee evaluation method and a system, which can be used in the field of biometrics, and can also be used in any field other than the field of biometrics. The application field of the facial expression recognition model training method, the employee evaluation method and the system in the present application is not limited.
[0038] In facial expression recognition, the identity information of the face may interfere with the accurate recognition of facial expression feature information by the facial expression recognition model, mainly due to the following reasons:
[0039] First, facial identity information and expression information may share certain features in an image. For example, some geometric features of the face (such as the shape of the mouth) are related to identity and will change when the expression changes. This feature overlap may cause the model to be interfered by identity features when recognizing expressions.
[0040] Second, differences in facial structure between individuals may cause the same expression to show different characteristics on different faces. For example, a smile may show different curvatures of the mouth corners or changes in the eyes on different faces. If the model focuses too much on identity features, it may misjudge these normal individual differences as different expressions.
[0041] Third, if the training data contains a lot of identity information, the model may tend to learn these identity features instead of focusing on expression features. This bias may lead to a decrease in the generalization ability of the model in recognizing expressions, especially when facing unseen individuals.
[0042] Fourth, expression recognition needs to focus on dynamic changes and subtle facial muscle movements, while identity information is usually static. Too much identity information may mask these dynamic features, making it difficult for the model to capture the details of expression changes.
[0043] Since the facial expression recognition model in the prior art usually directly extracts facial expressions from video frames, the identity information of the face in this process will interfere with the facial expression feature information, reducing the accuracy of the facial expression recognition model in recognizing facial expressions. Therefore, in response to the problems of the prior art, the technical concept of this application is: converting the facial slice image of the target frame of the sample video data into a feature map, then extracting facial feature information and facial identity information based on the feature map, removing the facial identity information from the facial feature information, and obtaining the first pure facial expression feature information; performing the same processing on the supporting frame of the sample video data to obtain the second pure facial expression feature information. Then, based on the first pure facial expression feature information and the second pure facial expression feature information, the loss value is calculated through the loss function, and the weight parameters of the facial expression recognition model are updated according to the loss value.
[0044] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0045] Figure 1 A flow chart of a facial expression recognition model training method provided in an embodiment of the present application, which can be applied to a server, such as Figure 1As shown, the method includes:
[0046] S101, performing a feature extraction operation on a facial slice image of a target frame of the sample video data to obtain first pure facial expression feature information, and performing a feature extraction operation on a facial slice image of a support frame of the sample video data to obtain second pure facial expression feature information, wherein the feature extraction operation comprises: converting the facial slice image into a feature map, extracting facial feature information and facial identity feature information in the feature map according to the feature map, removing the facial identity feature information from the facial feature information, and obtaining pure facial expression feature information;
[0047] Specifically, the supporting frames in the sample video data can be obtained from the instructional video shot by the trainer. A facial detection algorithm such as MTCNN, Dlib, etc. can be used to detect the facial area in each frame and crop the facial slice image. Then, a pre-trained convolutional neural network (CNN), such as VGG, ResNet, etc., can be used to input the facial slice image into the network and extract the output of the intermediate layer as the feature map.
[0048] In the step of extracting facial feature information and facial identity feature information, expression-related features may be extracted as facial feature information through a specific network layer, and identity-related features may be extracted as facial identity feature information through another network layer.
[0049] Next, adversarial training or feature separation techniques (such as variational autoencoders and adversarial neural networks) can be used to remove identity features. For example, a network structure can be designed in which one branch focuses on expression features and the other branch focuses on identity features, and the adversarial loss is used to minimize the impact of identity features on expression features.
[0050] S102, calculating a loss value through a loss function according to the first pure facial expression feature information and the second pure facial expression feature information, and updating a weight parameter of a facial expression recognition model according to the loss value.
[0051] Specifically, the gradient of the loss value can be calculated using the back propagation algorithm, and the weight parameters of the facial expression recognition model can be updated by gradient descent or its variants such as Adam, RMSprop, etc. The above process is integrated into the training loop, and the facial expression recognition model is continuously trained until the loss value converges or the preset number of training rounds is reached.
[0052] The facial expression recognition model training method provided in this embodiment can avoid the interference of identity information on facial expression feature information by removing identity information from facial feature information during the training stage, thereby improving the model's focus on dynamic changes in facial expressions and improving the model's accuracy in recognizing facial expressions.
[0053] This embodiment provides a possible facial expression recognition model, which is trained using the above method. Figure 2 A schematic diagram of the facial expression recognition model structure provided in the embodiment of the present application is shown in FIG. Figure 2 As shown in the figure, the specific workflow of this model in application is as follows:
[0054] First, use the Haar cascade classifier in OpenCV to perform face detection on N target frames and N support frames to obtain facial slice images. Similarly, the support frames come from the trainer's instruction video. Then adjust all facial slice images to a uniform size through the Resize operation to ensure the consistency of subsequent processing. For each facial slice image, a feature map is generated through a convolution operation, and then the feature map is transformed by wavelet to obtain N high-frequency components and N low-frequency components. Among them, the low-frequency component usually contains more identity information, and the high-frequency component usually contains less identity information.
[0055] The low-frequency components are processed by ResBlock three times to extract local features, and then processed by Transformer blocks (Vision Transformer) to obtain local correlation and generate low-frequency feature information. Since the high-frequency components have less identity information, they are directly processed by Vision Transformer blocks to generate high-frequency feature information.
[0056] Specifically, low-frequency components usually contain the overall structure and identity information of the face, which appear as larger areas and smoother changes in the image. ResBlock can be used to effectively capture these large-scale features and extract richer feature representations through deep networks. ResBlock can alleviate the gradient vanishing problem in deep networks by introducing residual connections, allowing the network to learn features more deeply and effectively. When processing low-frequency components, the Transformer block is used to capture local and global correlations. Since low-frequency information contains identity features, local correlations (such as the relationship between facial features) are important. Through the self-attention mechanism, the Transformer can flexibly model these correlations and enhance the ability of feature extraction.
[0057] High-frequency components mainly contain detail information, such as edges and textures, which are relatively unimportant in identity recognition. Using the Vision Transformer block to directly process high-frequency components can effectively capture these detail features. Vision Transformer uses the self-attention mechanism to model the relationship between features on a global scale, which is suitable for processing detail changes in images.
[0058] Next, two CNN layers are used to align the channel sizes of high-frequency feature information and low-frequency feature information, and then the high-frequency feature information and low-frequency feature information are re-fused through inverse wavelet transform. Next, the fused feature information is processed through orthogonal decomposition to remove the facial identity feature information from the facial feature information. Figure 2 As shown, the target frame is processed as above to obtain first pure facial expression feature information, and the support frame is processed as above to obtain second pure facial expression feature information.
[0059] The first clean facial expression feature information and the second clean facial expression feature information are processed using three Dense blocks respectively, the processed features are stacked according to the time dimension, and the similarity between the processed first clean facial expression feature information and the processed second clean facial expression feature information is calculated using 3D convolution to recognize facial expressions. It should be noted that Figure 2 In , the fully connected layers and the Softmax layers are only used for identity prediction, but these layers are not needed when the model is applied.
[0060] based on Figure 2 The facial expression recognition model shown, optionally, this embodiment provides a feature extraction operation method, Figure 3 A flow chart of a feature extraction operation method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes:
[0061] S301, performing wavelet transform on the feature map to generate high-frequency components and low-frequency components of the feature map;
[0062] S302, extracting high-frequency feature information of the high-frequency component and low-frequency feature information of the low-frequency component, and fusing the high-frequency feature information and the low-frequency feature information through inverse wavelet transform to obtain fused feature information;
[0063] S303, determining the impurity identity information in the fused feature information according to the fused feature information and the feature graph;
[0064] S304: remove the impure identity information from the fused feature information to obtain pure facial expression feature information.
[0065] Figure 3The feature extraction operation method shown can refer to the above-mentioned facial expression recognition model workflow, which will not be described here. Since the low-frequency component mainly contains the overall structure of the image and identity-related features, and the high-frequency component mainly contains detail information such as the edge and texture of the image, the feature extraction operation method can effectively combine the overall structure, details, and identity-related features through inverse wavelet transform, so that the impurity information related to the identity in the fused feature information can be identified by analyzing the fused feature information and the original feature map, and then these impurity identity information can be removed from the fused feature information to obtain pure facial expression feature information.
[0066] based on Figure 2 The facial expression recognition model shown, and Figure 3 The method shown, further optionally, determines the impurity identity information in the fused feature information according to the fused feature information and the feature map, including:
[0067] Through orthogonal decomposition, the projection of the fused feature information on the feature map is calculated, and the projection of the fused feature information on the feature map is used as the impurity identity information.
[0068] Specifically, orthogonal decomposition can be performed using the following formula:
[0069] A 分 =A 逆 -A 投
[0070]
[0071] Among them, A 逆 To fuse feature information, A 投 is the impurity identity information, A 原 is the feature map. Through orthogonal decomposition, the fused feature information can be decomposed into two orthogonal parts: a part aligned with the feature map (projection) and a part orthogonal to it. The projection of the fused feature information on the feature map is regarded as the impurity identity information, because the projection represents the part of the fused feature information that is most relevant to the feature map (usually containing identity information). Through orthogonal decomposition, the impurity identity information can be effectively identified and extracted.
[0072] like Figure 2 As shown in , the facial expression recognition model can be divided into an identity feature extraction neural network and a facial expression feature extraction neural network after the identity feature extraction network, where Figure 2 As shown, the identity feature extraction neural network includes the part between wavelet transform and inverse wavelet transform. The identity feature extraction neural network is specifically used for:
[0073] The feature map is subjected to wavelet transform to generate high-frequency components and low-frequency components of the feature map, and high-frequency feature information of the high-frequency components and low-frequency feature information of the low-frequency components are extracted. The high-frequency feature information and the low-frequency feature information are fused through inverse wavelet transform to obtain fused feature information.
[0074] Therefore, an alternating training strategy can be adopted for the identity feature extraction neural network and the facial expression feature extraction neural network. Optionally, according to the first pure facial expression feature information and the second pure facial expression feature information, a loss value is calculated by a loss function, and the weight parameters of the facial expression recognition model are updated according to the loss value, including:
[0075] Freeze the part after the identity feature extraction neural network of the facial expression recognition model, calculate the identity-related loss value through the identity-related loss function according to the fused feature information, and update the weight parameters of the identity feature extraction neural network according to the identity-related loss value;
[0076] Freeze the identity feature extraction neural network, calculate the expression-related loss value through the expression-related loss function according to the first pure facial expression feature information and the second pure facial expression feature information, and update the weight parameters of the facial expression feature extraction neural network according to the expression-related loss value.
[0077] Through the above steps, the identity feature extraction neural network and the facial expression feature extraction neural network are alternately trained until the identity-related loss value is less than the first loss threshold and the expression-related loss value is less than the second loss threshold.
[0078] The above alternating training method can accurately extract and recognize identity information by optimizing the identity feature extraction neural network alone, thereby ensuring that the identity features are effectively separated and processed in the subsequent expression recognition process; by freezing the identity feature extraction neural network, the model focuses on optimizing the expression features in the expression recognition stage without being disturbed by the identity features. By freezing and training different network parts in stages, the model can focus on a specific task (identity recognition or expression recognition) at each stage, thereby improving the model's performance in processing identity feature extraction and facial expression recognition tasks.
[0079] Optionally, the identity-related loss value is calculated by an identity-related loss function according to the fused feature information, including:
[0080] According to the fusion feature information, the cross entropy loss value and L1 loss value are calculated through the cross entropy loss function and the L1 loss function, and the weighted sum of the cross entropy loss value and the L1 loss value is used as the identity-related loss value. The steps for calculating the cross entropy loss value are as follows:
[0081] The fused feature information passes through the output of the fully connected layer and the Softmax layer (predicted probability distribution) and the true identity label, and then the cross entropy loss function is used to calculate the difference between the predicted probability distribution and the true identity label to obtain the cross entropy loss value. At the same time, the L1 loss function is used to calculate the L1 loss value between the fused feature information and the true identity label.
[0082] It should be noted that since cross entropy loss is often used in multi-class classification problems, it is particularly suitable for identity recognition tasks. It measures the difference between the predicted identity probability distribution and the true label; L1 loss is used to measure the absolute difference between feature vectors, thereby helping the model learn more recognizable identity features. By taking the weighted sum of cross entropy loss and L1 loss as the identity-related loss, the model can simultaneously optimize classification accuracy and feature recognition. This combined loss function can balance the impact of different losses and ensure that the model has higher accuracy and robustness in identity recognition tasks.
[0083] Optionally, calculating the expression-related loss value by using an expression-related loss function according to the first pure facial expression feature information and the second pure facial expression feature information includes:
[0084] The cosine similarity loss value between the first pure facial expression feature information and the second pure facial expression feature information is calculated by a cosine similarity loss function, and the cosine similarity loss value is used as an expression-related loss value.
[0085] It should be noted that by calculating the angular similarity between two feature vectors, cosine similarity is suitable for comparing facial expression features because it can ignore the amplitude changes of feature vectors and focus on the pattern and direction of features. Using cosine similarity loss to optimize the expression recognition part can ensure that the model can accurately recognize and match facial expressions in different video frames.
[0086] When the identity-related loss value is less than the first loss threshold, and the expression-related loss value is less than the second loss threshold, the identity feature extraction neural network and the facial expression feature extraction neural network can be unfrozen to fine-tune the model as a whole. Methods for fine-tuning the model as a whole include:
[0087] If the identity-related loss value is less than the first loss threshold and the expression-related loss value is less than the second loss threshold, the total loss value is calculated based on the first pure facial expression feature information and the second pure facial expression feature information through the identity-related loss function and the expression-related loss function, and the weight parameters of the facial expression recognition model are updated according to the total loss value until the total loss value is less than the third loss threshold.
[0088] Specifically, the above-mentioned cross entropy loss function, L1 loss function, and cosine similarity loss function can be used to calculate the weighted sum of cross entropy loss, L1 loss, and cosine similarity loss as the total loss value. This stage combines the losses of identity and expression to ensure that the model strikes a balance between the two. By calculating the total loss value (a combination of identity-related loss and expression-related loss), the model can simultaneously consider the optimization of identity and expression features during the fine-tuning stage. By using the total loss value in the fine-tuning stage, the model can simultaneously optimize on identity recognition and expression recognition tasks. This helps to improve the overall performance of the model, making it perform better when handling complex tasks.
[0089] Based on the facial expression recognition model trained by the above training method, this embodiment provides an employee evaluation method. Figure 4 A schematic diagram of a flow chart of an employee evaluation method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes:
[0090] S401, extracting first facial expression feature information of the candidate employee video data and second facial expression feature information of the first sample video data through a facial expression recognition model, and calculating expression similarity scores of the first facial expression feature information and the second facial expression feature information through the facial expression recognition model;
[0091] Specifically, the first sample video data can be a trainer's instruction video, and the candidate employee is the employee to be evaluated. The facial expression recognition model trained by the above training method is used to process the candidate employee video data. After removing the facial identity feature information by orthogonal decomposition, the first facial expression feature information is obtained. The first sample video data is processed in the same way to obtain the second facial expression feature information, and the similarity of the two feature information is calculated to obtain a score. It can be understood that the facial expression in the trainer's instruction video is a standard expression. The higher the similarity between the employee's expression and the standard expression, the more the employee's expression meets the standard.
[0092] S402, extracting first limb movement feature information of the candidate employee video data and second limb movement feature information of the second sample video data, and calculating a limb movement similarity score between the first limb movement feature information and the second limb movement feature information;
[0093] In this step, a body movement recognition model can be used to extract the first body movement feature information and the second body movement feature information of the second sample video data and calculate the body movement similarity score. This embodiment provides a possible body movement recognition model, Figure 5 A schematic diagram of a body movement recognition model structure provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the working principle of the body movement recognition model is as follows:
[0094] First, skeleton data is extracted from the video through library functions (such as OpenPose, MediaPipe, etc.). These libraries can detect key points of the human body (such as joint positions) and convert them into skeleton representations. This representation is usually two-dimensional or three-dimensional coordinate data that reflects the posture of the human body in each frame.
[0095] A self-attention mechanism is applied in each frame. This mechanism allows the model to consider the relevance of different positions in the input sequence when processing the input data. By calculating the attention weights between each keypoint and other keypoints, the model can better understand the relationship between different parts in each frame.
[0096] Three residual blocks (ResBlock) are used for preliminary feature extraction. ResBlock is a deep residual network structure that alleviates the gradient vanishing problem in deep networks by introducing skip connections. Each ResBlock usually contains a convolutional layer, a batch normalization layer, and a ReLU activation function. Through these layers, the model is able to extract richer feature representations.
[0097] Three Vision Transformer modules are used, stacked along the time dimension. By treating the skeleton data as a sequence input, the Vision Transformer can capture dynamic changes in the time dimension.
[0098] 3D convolution processes information in the time dimension. 3D convolution can perform convolution operations in both spatial and temporal dimensions, which enables the model to capture spatiotemporal features. This step can integrate information from different time frames to form a more representative feature vector.
[0099] The first limb movement feature information corresponding to the target video frame and the second limb movement feature information corresponding to the supporting frame of the second sample video data are obtained, and the model can quantify the similarity between the first limb movement feature information and the second limb movement feature information.
[0100] Finally, the calculated similarity is normalized. This step ensures that the similarity value is within a standard range (usually 0 to 1) to facilitate subsequent action recognition and classification.
[0101] It can be understood that the body movements in the trainer's instructional video are standard movements. The higher the similarity between the employee's movements and the standard movements, the more the employee's movements conform to the standards.
[0102] S403: Generate an evaluation result of the candidate employee based on the facial expression similarity score and the body movement similarity score.
[0103] Specifically, in this step, an evaluation model can be trained for evaluation. Since the facial expression recognition model and the body movement recognition model process multiple target frames and support frames and give similarity scores, the expression similarity score and the body movement similarity score can be similarity scores for multiple consecutive frames of the candidate employee video. Therefore, the evaluation model can give a machine evaluation result based on the expression similarity score and the body movement similarity score. The machine evaluation result may include, for example, the facial expression change trend, the expected expression, the expression fit, the body movement change trend, the expected body movement, and the body fit. By combining the expression similarity score and the body movement similarity score to generate the evaluation results of the candidate employee, the strengths and weaknesses of the candidate employee in the service process can be seen, and targeted improvements can be made in subsequent training.
[0104] Based on the above employee evaluation method, this embodiment provides an employee evaluation system. Figure 6 A schematic diagram of an employee evaluation system structure and workflow provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the system includes:
[0105] The facial expression recognition module 601 is used to: extract the first facial expression feature information of the candidate employee video data and the second facial expression feature information of the first sample video data through the facial expression recognition model, and calculate the expression similarity score of the first facial expression feature information and the second facial expression feature information through the facial expression recognition model;
[0106] The body movement analysis module 602 is used to extract first body movement feature information of the candidate employee video data and second body movement feature information of the second sample video data, and calculate a body movement similarity score between the first body movement feature information and the second body movement feature information;
[0107] The evaluation module 603 is used to generate an evaluation result of the candidate employee according to the expression similarity score and the body movement similarity score.
[0108] Figure 7 A schematic diagram of the structure of the facial expression recognition model training device provided in the embodiment of the present application, such as Figure 7 As shown, the device 70 includes:
[0109] The feature extraction module 701 is used to: perform a feature extraction operation on the facial slice image of the target frame of the sample video data to obtain first pure facial expression feature information, and perform a feature extraction operation on the facial slice image of the support frame of the sample video data to obtain second pure facial expression feature information, wherein the feature extraction operation includes: converting the facial slice image into a feature map, extracting facial feature information and facial identity feature information in the feature map according to the feature map, removing the facial identity feature information from the facial feature information, and obtaining pure facial expression feature information;
[0110] The weight updating module 702 is used to calculate the loss value through the loss function according to the first pure facial expression feature information and the second pure facial expression feature information, and update the weight parameters of the facial expression recognition model according to the loss value. The facial expression recognition model training device provided in the embodiment of the present application can execute the method provided in the above-mentioned facial expression recognition model training method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.
[0111] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.
[0112] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.
[0113] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0114] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0115] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.
[0116] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0117] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0118] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0119] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0120] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0121] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0122] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0123] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0124] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0125] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0126] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A facial expression recognition model training method, characterized in that: The method comprises: Performing a feature extraction operation on a facial slice image of a target frame of sample video data to obtain first pure facial expression feature information, and performing the feature extraction operation on a facial slice image of a support frame of the sample video data to obtain second pure facial expression feature information, wherein the feature extraction operation includes: converting the facial slice image into a feature map, extracting facial feature information and facial identity feature information in the feature map according to the feature map, removing the facial identity feature information from the facial feature information, and obtaining pure facial expression feature information; According to the first pure facial expression feature information and the second pure facial expression feature information, a loss value is calculated by a loss function, and a weight parameter of the facial expression recognition model is updated according to the loss value.
2. The method according to claim 1, characterized in that The step of converting the facial slice image into a feature map, extracting facial feature information and facial identity feature information in the feature map according to the feature map, removing the facial identity feature information from the facial feature information, and obtaining pure facial expression feature information includes: Performing wavelet transformation on the feature map to generate high-frequency components and low-frequency components of the feature map; Extracting high-frequency feature information of the high-frequency component and low-frequency feature information of the low-frequency component, and fusing the high-frequency feature information and the low-frequency feature information through inverse wavelet transform to obtain fused feature information; Determining impurity identity information in the fused feature information according to the fused feature information and the feature graph; The impurity identity information is removed from the fused feature information to obtain the pure facial expression feature information.
3. The method according to claim 2, characterized in that The step of determining the impurity identity information in the fused feature information according to the fused feature information and the feature graph includes: The projection of the fused feature information on the feature map is calculated through orthogonal decomposition, and the projection of the fused feature information on the feature map is used as the impurity identity information.
4. The method according to claim 1, characterized in that: The step of calculating a loss value through a loss function according to the first pure facial expression feature information and the second pure facial expression feature information, and updating a weight parameter of the facial expression recognition model according to the loss value comprises: Freezing the part after the identity feature extraction neural network of the facial expression recognition model, wherein the identity feature extraction neural network is used to: perform wavelet transform on the feature map to generate high-frequency components and low-frequency components of the feature map, extract high-frequency feature information of the high-frequency components and low-frequency feature information of the low-frequency components, and fuse the high-frequency feature information and the low-frequency feature information through inverse wavelet transform to obtain fused feature information; Calculating an identity-related loss value through an identity-related loss function according to the fused feature information, and updating a weight parameter of the identity feature extraction neural network according to the identity-related loss value until the identity-related loss value is less than a first loss threshold; Freeze the identity feature extraction neural network, calculate the expression-related loss value through the expression-related loss function according to the first pure facial expression feature information and the second pure facial expression feature information, and update the weight parameters of other parts of the facial expression recognition model except the identity feature extraction neural network according to the expression-related loss value until the expression-related loss value is less than the second loss threshold.
5. The method according to claim 4, characterized in that The calculating the identity-related loss value by using the identity-related loss function according to the fused feature information includes: According to the fused feature information, a cross entropy loss value and an L1 loss value are calculated by using a cross entropy loss function and an L1 loss function, and a weighted sum of the cross entropy loss value and the L1 loss value is used as the identity-related loss value; The step of calculating the expression-related loss value by using an expression-related loss function according to the first pure facial expression feature information and the second pure facial expression feature information comprises: The cosine similarity loss value between the first clean facial expression feature information and the second clean facial expression feature information is calculated by using a cosine similarity loss function, and the cosine similarity loss value is used as the expression-related loss value.
6. The method according to claim 4, characterized in that The method further comprises: If the identity-related loss value is less than the first loss threshold, and the expression-related loss value is less than the second loss threshold, the total loss value is calculated according to the first pure facial expression feature information and the second pure facial expression feature information through the identity-related loss function and the expression-related loss function, and the weight parameters of the facial expression recognition model are updated according to the total loss value until the total loss value is less than the third loss threshold.
7. A method for employee evaluation, characterized in that: The employee evaluation method is implemented based on a facial expression recognition model, wherein the facial expression recognition model is trained by the method according to any one of claims 1 to 6, and the employee evaluation method comprises: Extracting first facial expression feature information of the candidate employee video data and second facial expression feature information of the first sample video data through the facial expression recognition model, and calculating expression similarity scores of the first facial expression feature information and the second facial expression feature information through the facial expression recognition model; Extracting first limb movement feature information of the candidate employee video data and second limb movement feature information of the second sample video data, and calculating a limb movement similarity score between the first limb movement feature information and the second limb movement feature information; An evaluation result of the candidate employee is generated according to the facial expression similarity score and the body movement similarity score.
8. An employee evaluation system, characterized in that: The system is implemented based on a facial expression recognition model, and the facial expression recognition model is trained by the method according to any one of claims 1 to 6. The system comprises: A facial expression recognition module, used to: extract first facial expression feature information of the candidate employee video data and second facial expression feature information of the first sample video data through the facial expression recognition model, and calculate the expression similarity score of the first facial expression feature information and the second facial expression feature information through the facial expression recognition model; A body movement analysis module is used to: extract first body movement feature information of the candidate employee video data and second body movement feature information of the second sample video data, and calculate a body movement similarity score between the first body movement feature information and the second body movement feature information; An evaluation module is used to generate an evaluation result of the candidate employee according to the expression similarity score and the body movement similarity score.
9. A facial expression recognition model training device, characterized in that: The device comprises: A feature extraction module is used to: perform a feature extraction operation on a facial slice image of a target frame of sample video data to obtain first pure facial expression feature information, and perform the feature extraction operation on a facial slice image of a support frame of the sample video data to obtain second pure facial expression feature information, wherein the feature extraction operation includes: converting the facial slice image into a feature map, extracting facial feature information and facial identity feature information in the feature map according to the feature map, removing the facial identity feature information from the facial feature information, and obtaining pure facial expression feature information; A weight updating module is used to calculate a loss value through a loss function according to the first pure facial expression feature information and the second pure facial expression feature information, and update a weight parameter of the facial expression recognition model according to the loss value.
10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6, or the method according to claim 7.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 6, or the method according to claim 7.
12. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6, or the method according to claim 7.