Facial expression-based sentiment analysis method

Abstract

The invention discloses a facial expression-based sentiment analysis method, relates to the technical field of sentiment analysis in the power industry, and aims to improve the sentiment monitoring and safety management efficiency of power grid workers. The method comprises the following steps: acquiring facial expressions by using a camera, and performing face detection by using an RFB-320 model; using a convolutional neural network to extract features and classify emotions; the method combines a YOLOv5 model to perform sentiment analysis and evaluation, identifies the emotional state of the employee, improves the generalization ability and robustness of the model through data enhancement and optimization strategies, overcomes the problems of detection environment limitation, real-time performance and efficiency in the prior art, simplifies the model complexity and deployment difficulty, and improves the detection efficiency. Fine granularity of emotion classification and accuracy of analysis are enhanced, hardware cost is reduced, the system is suitable for various working environments, real-time monitoring and feedback of employee emotion are achieved through an integrated emotion analysis system, management suggestions are provided, and safety management of power grid workers is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion analysis of power industry employees, and in particular to an emotion analysis method based on facial expressions. Background Art

[0002] As an important pillar of the national economy, the power industry has extremely high requirements for safety management. With the continuous development of the power grid industry and the application of artificial intelligence technology, facial expression recognition technology has gradually attracted people's attention as a new security means. This technology analyzes employees' facial expressions and monitors their emotional state in real time, so as to detect abnormal situations in time and take corresponding measures, thereby improving the efficiency and accuracy of safety management of power grid workers. In order to further improve the level of power grid safety management, we decided to introduce facial expression recognition technology to meet the needs of personnel safety management.

[0003] The prior art CN110751015A provides a method for perfusion optimization and artificial intelligence emotion monitoring of facial infrared thermal images, wherein, based on the facial infrared video of the monitoring target individual, the CNN-BiGRU_Attention network model is used to process the blood perfusion pseudo-color image corresponding to each frame of the facial infrared image, and then the spatial feature vector matrix contained in the frame is mined; in addition, the temporal feature vector matrix between frames of the facial infrared video is determined by the BiGRU layer; finally, based on the spatial feature vector matrix corresponding to each frame of the facial infrared image and the temporal feature vector matrix between adjacent frames, the probability feature information of the monitoring target individual for each preset emotional state is determined. The above technical solution combines CNN with BiGRU, and the BiGRU module introduces the Attention mechanism to fully extract the spatial features and inter-frame temporal features of the image, and then jointly identify the probability feature information of the monitoring target individual for each preset emotional state, so that a more realistic emotional distribution can be obtained in a non-interfering test environment, effectively improving the efficiency and accuracy of emotion recognition.

[0004] Although the above-mentioned existing technologies have made some progress in facial expression recognition and emotion analysis, they still have some shortcomings and defects when applied to emotion monitoring and safety management of power grid workers: 1. Limitations of the detection environment: Existing technologies (such as CN110751015A) mainly rely on facial infrared thermal images and complex deep learning models (such as CNN-BiGRU_Attention) for emotion recognition, which may be effective in certain specific scenarios, but in actual applications, the network structure is relatively complex and the computational cost is high, and high-performance hardware support may be required. At the same time, the acquisition and processing equipment of infrared thermal images is expensive and has high requirements for the environment. The performance of infrared images under different lighting conditions may be inconsistent, affecting the accuracy of sentiment analysis. It is not suitable for popular use in the complex environment where power grid workers work on a daily basis.

[0005] 2. Real-time and efficiency issues: Existing technologies usually involve a lot of calculations and data processing in the emotion recognition process, especially the extraction and analysis of time features of multiple frames of images, which will affect the real-time and efficiency of the system. In the actual management of power grid staff, the emotional monitoring of power grid staff requires efficient and fast real-time processing capabilities, and requires real-time monitoring and analysis of the emotional state of employees in order to take necessary management measures in a timely manner. Complex models may not meet the real-time requirements.

[0006] 3. Model complexity and deployment difficulty: Existing deep learning models such as the BiGRU_Attention network model have complex structures, high training and deployment costs, and require a large amount of computing resources and professional technicians for maintenance and optimization. This makes it difficult to implement and promote it widely in actual applications in the power grid industry, especially in some relatively remote working environments.

[0007] 4. Insufficient granularity of sentiment classification: The categories of sentiment classification in existing technologies are relatively simple and cannot fully reflect the specific emotional state and emotional change trend of employees. In the power grid industry, more fine-grained sentiment classification and analysis can help managers understand the emotional state of employees more accurately and take effective management measures in a timely manner.

[0008] 5. Insufficient generalization and robustness: Existing technologies are insufficient in the generalization and robustness of emotion recognition. They are mainly tested in laboratories or controlled environments, and their performance in actual applications may be affected by various factors, such as lighting changes, camera angles, employee facial occlusion, etc. In the field environment of the power grid, these factors are more complex and changeable, which puts higher requirements on the robustness of the emotion analysis system. The emotion recognition model needs to deal with various complex environmental changes and individual differences of employees, including lighting changes, angle changes, and individual expression differences. In these aspects, the robustness and generalization of existing technologies need to be improved to ensure stability and accuracy in different environments and conditions. Summary of the invention

[0009] The present invention aims to solve the problems of detection environment limitations, real-time and efficiency issues, model complexity and deployment difficulty, and insufficient generalization and robustness of emotion classification in the process of power grid staff management in the prior art. By analyzing and identifying employees' facial expressions in real time, accurately judging the emotional state of employees, and taking corresponding management measures in a timely manner, the efficiency and safety of power grid personnel management are improved, providing a more efficient, adaptable and easy-to-deploy emotion analysis method based on facial expressions to improve the efficiency and safety of power grid personnel management.

[0010] In order to solve the above problems, the present invention provides a method for emotion analysis based on facial expression, comprising the following steps: S1. Facial expression acquisition: Use a camera or other image acquisition equipment to collect the facial expressions of power grid workers in real time, and use the RFB-320 model to perform face detection on the input frame; S2. Expression recognition and analysis: Using computer vision and deep learning algorithms, we can recognize and analyze the expressions of collected facial images to identify the employees’ happy, sad or angry emotional states; S3. Emotional analysis and evaluation: Based on the recognized facial expressions, analyze and evaluate the employee’s emotions to determine the employee’s current emotional state and emotional tendency; S4. Emotion monitoring and feedback: Integrate the results of sentiment analysis with the employee management system to achieve real-time monitoring and feedback of employees’ emotional status, promptly notify relevant managers, and provide corresponding management suggestions and measures.

[0011] Preferably, in the facial expression acquisition step, face detection specifically includes the following steps: a. Initialize important parameters: image_mean: the mean of image normalization on RGB channels; image_std: standard deviation of image normalization; iou_threshold: determines the threshold of the intersection-over-union (IoU) metric for bounding box matching; center_variance: scaling factor of the predicted bounding box center coordinates; size_variance: scaling factor for the predicted bounding box size; min_boxes: minimum bounding box size for objects of different sizes; strides: controls the scale of the feature map according to the image size; threshold: confidence threshold for target detection; b. Call define_img_size function to generate a priori bounding box and calculate the size of feature map according to the provided image size and predefined step value; c. Use the RFB-320 model to process the input frame and obtain the bounding box for face detection.

[0012] Preferably, in the expression recognition and analysis step, a convolutional neural network is used to extract features of facial images and perform emotion classification, and a custom VGG13 model architecture is used to perform expression recognition and analysis on the detected face, including the following steps: a. Resize the detected facial image to a 64×64 grayscale image; b. Classify the images into seven emotion categories using convolutional layers with max pooling and dropout; c. Aggregate information through two dense layers with 1024 nodes and finally predict the emotion category through a softmax output layer.

[0013] Preferably, the convolutional neural network extracts features of facial images and performs emotion classification, specifically comprising the following steps: a. The first convolutional layer uses 64 convolution kernels of size 5×5, a stride of 1, ReLU as the activation function, the convolution mode is "same", and the input tensor size is 48×48×3; b. The first pooling layer: the pooling kernel size is 2×2 and the stride is 2; c. Batch normalization layer and Dropout layer that randomly discards 40% of network connections to prevent overfitting; d. The second convolutional layer uses 128 convolution kernels of size 5×5, a stride of 1, ReLU as the activation function, and the convolution mode is "same"; e. Second pooling layer: the pooling kernel size is 2×2, the stride is 2, followed by batch normalization and Dropout operations; f. The third convolutional layer: uses 256 convolution kernels of size 5×5, a stride of 1, uses ReLU as the activation function, and the convolution mode is "same"; g. The third pooling layer: the pooling kernel size is 2×2, and the stride is 2; h. Fully connected layer: contains 2048 neurons, the activation function is ReLU, and 40% of the network connections are randomly discarded; i. The second fully connected layer: contains 1024 neurons, the activation function is ReLU, and 40% of the network connections are randomly discarded; j. The third fully connected layer: contains 512 neurons, and the activation function is ReLU; k. Classification output layer: contains 7 output neurons and uses softmax as the activation function for sentiment classification.

[0014] Preferably, in the sentiment analysis and evaluation step, a YOLOv5 model is used for target detection, and the model includes the following modules: a. Simplified convolutional layer: Contains convolution operation, batch normalization and ReLU activation function, convolution kernel size is 3, stride is 1, and automatic padding makes the input and output size the same; b. CSPNet module: It includes 1×1 convolution of the number of segmented channels, multi-layer 3×3 convolution and the final 1×1 convolution. It enhances the feature representation capability by splicing multiple intermediate results. c. YOLOv5 Backbone: contains multiple convolutional layers and CSP modules to extract features layer by layer. The beginning of the network contains a 3×3 convolutional layer, followed by multiple CSP modules to increase the number of channels and feature map size layer by layer. d. YOLOv5 Head: Contains a 1×1 convolutional layer and a final detection layer that predicts the category and bounding box for each pixel.

[0015] Preferably, in the sentiment analysis and evaluation step, a machine learning or deep learning method is used to train a sentiment classification model for matching and classifying the recognized facial expressions with sentiment categories, and the sentiment classification model includes a support vector machine (SVM) or a decision tree or a neural network.

[0016] Preferably, the sentiment analysis and evaluation step also includes feature extraction and representation, and the method of feature extraction and representation includes converting the facial image into a feature vector that can be processed by a classification algorithm, and the features include local binary pattern (LBP), principal component analysis (PCA), and face embedding.

[0017] Preferably, the optimization of the training of the emotion classification model includes collecting a large number of labeled facial expression data sets, using these data to train the emotion classification model, and optimizing the model through methods such as cross-validation to improve the generalization ability and accuracy of the model.

[0018] Preferably, the training process of the sentiment classification model specifically includes: S1. Data enhancement: Perform data enhancement operations such as rotation, cropping, mirroring, and brightness adjustment on the training data to improve the robustness of the model; S2. Loss function: Use the cross entropy loss function to measure the gap between the model prediction results and the actual labels; S3. Optimization algorithm: Adopt adaptive learning rate optimization algorithm to update model parameters; S4. Learning rate adjustment: dynamically adjust the learning rate according to the loss changes during training, using a learning rate decay strategy; S5. Model evaluation: Perform model evaluation through validation sets to monitor performance changes of the model during training and prevent overfitting.

[0019] The training of the sentiment classification model can also be performed by integrating multiple sub-models to perform sentiment analysis to improve classification accuracy and robustness. The specific steps include: a. Sub-model integration: train multiple models with different structures or parameters, and improve the overall performance through integrated learning methods (Bagging or Boosting); b. Model fusion: Use weighted average and voting methods to fuse the prediction results of multiple models to obtain the final sentiment classification result.

[0020] Preferably, the hardware architecture of the system for emotion analysis based on facial expressions includes: High-performance image acquisition equipment: used to collect employee facial images; Computing resources: high-performance servers or cloud computing resources to run facial expression recognition and sentiment analysis algorithms; Storage device: used to store collected image data and analysis results; Network communication equipment: used to achieve efficient communication between data collection, transmission and processing; The software architecture of the system includes: Data collection module: responsible for collecting and preprocessing employees’ facial images; Expression recognition module: Use pre-trained deep learning models to perform expression recognition on facial images; Sentiment analysis module: sentiment classification and analysis based on expression recognition results; Result feedback module: Feedback the sentiment analysis results to the management system and generate corresponding reports and suggestions; System management module: used for system configuration, monitoring and maintenance.

[0021] Compared with the prior art, the present invention has the following beneficial technical effects: Enhanced adaptability to the testing environment: The present invention uses an RGB camera and RFB-320 (SSD) model to collect facial expressions, avoiding dependence on infrared thermal images and reducing sensitivity to ambient lighting conditions. By using a convolutional neural network to extract the features of facial images and high-efficiency target detection capabilities, it can work stably under a variety of lighting conditions, significantly improving the applicability of emotion analysis in different working environments. This method reduces hardware costs and environmental requirements, making facial expression collection equipment more popular and suitable for the complex environment of daily work of power grid workers.

[0022] Real-time performance and efficiency improvement: The present invention adopts an optimized convolutional neural network structure for target detection. The convolutional neural network structure can achieve rapid face detection and expression recognition while ensuring accuracy. It further improves the speed and efficiency of emotion classification, meets the real-time requirements of emotion monitoring of power grid workers, and combines YOLOv5 to analyze and classify facial expressions, greatly simplifying the calculation process. The present invention adopts a combination of YOLOv5 and convolutional neural network. Compared with complex network structures such as BiGRU_Attention, the model structure of the present invention is simpler, the calculation speed is faster, the computing resource requirements are lower, and it is easy to deploy and maintain in the field environment. It can meet the real-time requirements of emotion monitoring of power grid workers and ensure that abnormal emotions are responded to in the first time.

[0023] 3. Model complexity and deployment simplification: This invention adopts a combination of YOLOv5 and convolutional neural network. Compared with complex network structures such as BiGRU_Attention, the model design is simpler, the computing resource requirements are lower, and it is easy to deploy and maintain in the field environment.

[0024] 4. Sentiment classification and analysis fine-grained enhancement: The present invention can identify more delicate emotional states through fine expression feature extraction and analysis. Compared with the simple emotion classification in the prior art, the emotion classification of the present invention is richer and more accurate, which can help managers understand the emotional state of employees more comprehensively. By combining the precise positioning capability of YOLOv5 and the powerful feature extraction capability of convolutional neural networks, it is possible to perform fine-grained analysis of facial expressions and provide more detailed emotional information.

[0025] The present invention uses convolutional neural networks for sentiment classification, which can refine and classify employees' specific emotional states, including seven emotional categories such as happiness, sadness, and anger, and provide more detailed and accurate sentiment analysis. Through sentiment analysis and evaluation steps, the YOLOv5 model is used to conduct an in-depth assessment of employees' emotions, helping managers to more accurately understand employees' emotional states and take effective management measures in a timely manner.

[0026] 5. Improved generalization and robustness: The present invention fully considers the variability and complexity of the power grid field environment. The convolutional neural network can maintain a high emotion recognition accuracy rate under different lighting conditions, camera angles and face occlusions through efficient feature extraction and region detection mechanisms combined with the powerful learning ability of the YOLOv5 model. In particular, through data enhancement and model optimization strategies, the emotion analysis system has strong robustness and generalization capabilities in various complex environments, ensuring its stability and reliability in practical applications.

[0027] The present invention adopts a variety of data enhancement technologies and optimization strategies, such as rotation, cropping, mirroring, etc., to train and optimize the model, which significantly improves the generalization ability and robustness of the emotion recognition model.

[0028] The integrated multi-model training method further improves the classification accuracy and robustness, making emotion recognition stable in different environments and conditions.

[0029] In summary, the present invention significantly improves the environmental adaptability, real-time performance, accuracy and robustness of facial expression emotion analysis by combining YOLOv5 and convolutional neural network, while simplifying the model complexity and deployment difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of the emotion analysis method based on facial expression of the present invention.

[0031] Figure 2 A diagram of the VGG13 model architecture customized for convolutional neural networks.

[0032] Figure 3 This is the algorithm model diagram of the convolutional neural network.

[0033] Figure 4 This is the algorithm model diagram of YOLOv5. DETAILED DESCRIPTION

[0034] The present invention will be further explained below in conjunction with the accompanying drawings and embodiments.

[0035] Reference Figure 1 As shown, the present invention provides a method for emotion analysis based on facial expression, comprising the following steps: S1. Facial expression acquisition: Use a camera or other image acquisition equipment to collect the facial expressions of power grid workers in real time, and use the RFB-320 model to perform face detection on the input frame; S2. Expression recognition and analysis: Using computer vision and deep learning algorithms, we can recognize and analyze the expressions of collected facial images to identify the employees’ happy, sad or angry emotional states; S3. Emotional analysis and evaluation: Based on the recognized facial expressions, analyze and evaluate the employee’s emotions to determine the employee’s current emotional state and emotional tendency; S4. Emotion monitoring and feedback: Integrate the results of sentiment analysis with the employee management system to achieve real-time monitoring and feedback of employees’ emotional status, promptly notify relevant managers, and provide corresponding management suggestions and measures.

[0036] In the facial expression collection step, a. Initialize important parameters: image_mean: the mean of image normalization on RGB channels; image_std: standard deviation of image normalization; iou_threshold: determines the threshold of the intersection-over-union (IoU) metric for bounding box matching; center_variance: scaling factor of the predicted bounding box center coordinates; size_variance: scaling factor for the predicted bounding box size; min_boxes: minimum bounding box size for objects of different sizes; strides: controls the scale of the feature map according to the image size; threshold: confidence threshold for target detection; b. Call define_img_size function to generate a priori bounding box and calculate the size of feature map according to the provided image size and predefined step value; c. Use the RFB-320 model to process the input frame and obtain the bounding box for face detection.

[0037] The convolutional neural network uses a custom VGG13 model architecture. The emotion recognition classification model uses the VGG13 architecture, which is designed for 64×64 grayscale images. The architecture uses convolutional layers with maximum pooling and dropout to classify images into seven emotion categories to prevent overfitting. The specific structure is as follows: 1. The initial part is two convolutional layers with 64 convolution kernels, followed by maximum pooling and 25% dropout.

[0038] 2. Additional convolutional layers are used to capture complex features.

[0039] 3. Two dense layers with 1024 nodes are used to aggregate information, followed by 50% dropout.

[0040] 4. Finally, the emotion category is predicted through the softmax output layer.

[0041] Face detection specifically includes the following steps: 1. Initialization parameters During the implementation, the following important parameters need to be initialized: - `image_mean`: Normalize the image by the mean across the RGB channels.

[0042] - `image_std`: Standard deviation of the image normalization.

[0043] - `iou_threshold`: Determines the threshold for the Intersection over Union (IoU) metric for bounding box matching.

[0044] - `center_variance`: Scaling factor for the predicted bounding box center coordinates.

[0045] - `size_variance`: Scaling factor for predicted bounding box size.

[0046] - `min_boxes`: Minimum bounding box sizes for objects of different sizes.

[0047] - `strides`: Control the scale of feature maps according to image size.

[0048] - `threshold`: confidence threshold for object detection.

[0049] 2. Prior Bounding Box Generation The `define_img_size` function is used to generate prior bounding boxes for the object detection task. This function takes the `image_size` parameter as input and calculates the size of the feature map based on the provided image size and a predefined stride value to achieve the desired output size for different input image scales. The prior bounding boxes in SSD provide an efficient way to simultaneously predict multiple bounding boxes and their associated class scores in a single forward pass of the network, thereby achieving real-time object detection.

[0050] 3. Real-time facial emotion recognition Use the `FER_live_cam` function to perform real-time facial emotion recognition on video frames. The steps are as follows: 1. Initialize a dictionary `emotion_dict` that maps numeric emotion category indices to human-readable emotion labels.

[0051] 2. Initialize a video source, you can use a local video file or a webcam input, and initialize an output video writer to save the processed frames with emotion annotations.

[0052] 3. The main emotion prediction model is saved in ONNX format and read using OpenCVDNN’s `readNetFromONNX` method, and loaded together with the RFB-320SSD face detection model saved in Caffe format.

[0053] 4. When processing the video frame by frame, perform face detection through the RFB-320 SSD model and obtain the bounding box.

[0054] 5. Preprocess the detected faces, including resizing and converting them to grayscale images, and then input them into the emotion recognition model for emotion prediction.

[0055] 6. Determine the recognized emotion by selecting the maximum value from the model’s output scores and map it to a label using emotion_dict.

[0056] 7. Add a rectangular box and emotion label around the detected face, save the frame to the output video file, and display it in real time. The user can stop the video display by pressing the 'q' key.

[0057] 8. Once video processing is completed or interrupted, release resources such as video capture and writers, and close any open windows.

[0058] Reference Figure 3 As shown in the figure, the convolutional neural network model includes the following key steps: 1. The first convolutional layer uses 64 convolution kernels of size 5×5, a stride of 1, ReLU as the activation function, the convolution mode is "same", and the input tensor size is 48×48×3.

[0059] 2. In the first pooling layer, the pooling kernel size is 2×2 and the stride is 2.

[0060] 3. Batch normalization layer and Dropout layer that randomly discards 40% of the network connections to prevent overfitting.

[0061] 4. The second convolutional layer uses 128 convolution kernels of size 5×5, a stride of 1, ReLU as the activation function, and the convolution mode is "same".

[0062] 5. In the second pooling layer, the pooling kernel size is 2×2 and the stride is 2, followed by batch normalization and Dropout operations.

[0063] 6. The third convolutional layer uses 256 convolution kernels of size 5×5, a stride of 1, ReLU as the activation function, and the convolution mode is "same".

[0064] 7. In the third pooling layer, the pooling kernel size is 2×2 and the stride is 2.

[0065] 8. Flatten the feature map into a one-dimensional vector and classify it through a fully connected layer.

[0066] 9. Fully connected layer, containing 2048 neurons, the activation function is ReLU, and 40% of the network connections are randomly dropped.

[0067] 10. The second fully connected layer contains 1024 neurons, the activation function is ReLU, and 40% of the network connections are randomly dropped.

[0068] 11. The third fully connected layer contains 512 neurons and the activation function is ReLU.

[0069] 12. The last layer is the classification output layer, which contains 7 output neurons and uses softmax as the activation function for sentiment classification.

[0070] In the emotion analysis and evaluation step, a machine learning or deep learning method is used to train an emotion classification model for matching and classifying the recognized facial expressions with emotion categories. The emotion classification model includes a support vector machine (SVM) or a decision tree or a neural network.

[0071] The sentiment analysis and evaluation step also includes feature extraction and representation, and the method of feature extraction and representation includes converting the facial image into a feature vector that can be processed by the classification algorithm, and the features include local binary pattern (LBP), principal component analysis (PCA), and face embedding.

[0072] The optimization of the training of the emotion classification model includes collecting a large number of labeled facial expression data sets, using these data to train the emotion classification model, and optimizing the model through methods such as cross-validation to improve the generalization ability and accuracy of the model.

[0073] The training process of the sentiment classification model specifically includes: S1. Data enhancement: Perform data enhancement operations such as rotation, cropping, mirroring, and brightness adjustment on the training data to improve the robustness of the model; S2. Loss function: Use the cross entropy loss function to measure the gap between the model prediction results and the actual labels; S3. Optimization algorithm: Adopt adaptive learning rate optimization algorithm to update model parameters; S4. Learning rate adjustment: dynamically adjust the learning rate according to the loss changes during training, using a learning rate decay strategy; S5. Model evaluation: Perform model evaluation through validation sets to monitor performance changes of the model during training and prevent overfitting.

[0074] The training of the sentiment classification model can also be performed by integrating multiple sub-models to perform sentiment analysis to improve classification accuracy and robustness. The specific steps include: a. Sub-model integration: train multiple models with different structures or parameters, and improve the overall performance through integrated learning methods (Bagging or Boosting); b. Model fusion: Use weighted average and voting methods to fuse the prediction results of multiple models to obtain the final sentiment classification result.

[0075] Preferably, the hardware architecture of the system for emotion analysis based on facial expressions includes: High-performance image acquisition equipment: used to collect employee facial images; Computing resources: high-performance servers or cloud computing resources to run facial expression recognition and sentiment analysis algorithms; Storage device: used to store collected image data and analysis results; Network communication equipment: used to achieve efficient communication between data collection, transmission and processing; The software architecture of the system includes: Data collection module: responsible for collecting and preprocessing employees’ facial images; Expression recognition module: Use pre-trained deep learning models to perform expression recognition on facial images; Sentiment analysis module: sentiment classification and analysis based on expression recognition results; Result feedback module: Feedback the sentiment analysis results to the management system and generate corresponding reports and suggestions; System management module: used for system configuration, monitoring and maintenance.

[0076] Reference Figure 4 As shown, the YOLOv5 model includes the following key steps: 1. Simplified convolution layer: Contains convolution operation, batch normalization and ReLU activation function, convolution kernel size is 3, stride is 1, and automatic padding makes the input and output sizes the same.

[0077] 2. CSPNet module: Contains 1×1 convolution of the number of segmented channels, multi-layer 3×3 convolution, and the final 1×1 convolution. By splicing multiple intermediate results, the feature representation capability is enhanced.

[0078] 3. YOLOv5 Backbone: Contains multiple convolutional layers and CSP modules to extract features layer by layer. The beginning of the network contains a 3×3 convolutional layer, followed by multiple CSP modules, which increase the number of channels and feature map size layer by layer.

[0079] 4. YOLOv5 Head: Contains a 1×1 convolutional layer and a final detection layer that predicts the category and bounding box of each pixel.

[0080] In the description of the present invention, it should be noted that the terms "upper", "lower", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0081] It should be noted that, in the present invention, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0082] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features of the invention described herein.

Claims

1. A method for emotion analysis based on facial expression, characterized in that: The following steps are involved: S1, facial expression acquisition: Use a camera or other image acquisition equipment to collect the facial expressions of power grid workers in real time, and use the RFB-320 (SSD) model to perform face detection on the input frame; S2. Expression recognition and analysis: Using computer vision and deep learning algorithms, we can recognize and analyze the expressions of collected facial images to identify the employees’ happy, sad or angry emotional states; S3. Emotional analysis and evaluation: Based on the recognized facial expressions, analyze and evaluate the employee’s emotions to determine the employee’s current emotional state and emotional tendency; S4. Emotion monitoring and feedback: Integrate the results of sentiment analysis with the employee management system to achieve real-time monitoring and feedback of employees’ emotional status, promptly notify relevant managers, and provide corresponding management suggestions and measures.

2. The method for emotion analysis based on facial expression according to claim 1, characterized in that: In the facial expression collection step, face detection specifically includes the following steps: a. Initialize important parameters: image_mean: the mean of image normalization on RGB channels; image_std: standard deviation of image normalization; iou_threshold: determines the threshold of the intersection-over-union (IoU) metric for bounding box matching; center_variance: scaling factor of the predicted bounding box center coordinates; size_variance: scaling factor for the predicted bounding box size; min_boxes: minimum bounding box size for objects of different sizes; strides: controls the scale of the feature map according to the image size; threshold: confidence threshold for target detection; b. Call define_img_size function to generate a priori bounding box and calculate the size of feature map according to the provided image size and predefined step value; c. Use the RFB-320 model to process the input frame and obtain the bounding box for face detection.

3. The method for emotion analysis based on facial expression according to claim 1, characterized in that: In the expression recognition and analysis step, a convolutional neural network is used to extract features of facial images and perform emotion classification, and a custom VGG13 model architecture is used to perform expression recognition and analysis on the detected face, including the following steps: a. Resize the detected facial image to a 64×64 grayscale image; b. Classify the images into seven emotion categories using convolutional layers with max pooling and dropout; c. Aggregate information through two dense layers with 1024 nodes and finally predict the emotion category through a softmax output layer.

4. The method for emotion analysis based on facial expression according to claim 3, characterized in that: The convolutional neural network extracts features of facial images and performs emotion classification, specifically including the following steps: a. The first convolutional layer: uses 64 convolution kernels of size 5×5, a stride of 1, uses ReLU as the activation function, the convolution mode is "same", and the input tensor size is 48×48×3; b. The first pooling layer: the pooling kernel size is 2×2 and the stride is 2; c. Batch normalization layer and Dropout layer that randomly discards 40% of network connections to prevent overfitting; d. The second convolutional layer uses 128 convolution kernels of size 5×5, a stride of 1, ReLU as the activation function, and the convolution mode is "same"; e. Second pooling layer: the pooling kernel size is 2×2, the stride is 2, followed by batch normalization and Dropout operations; f. The third convolutional layer: uses 256 convolution kernels of size 5×5, a step size of 1, uses ReLU as the activation function, and the convolution mode is "same"; g. The third pooling layer: the pooling kernel size is 2×2, and the stride is 2; h. Fully connected layer: contains 2048 neurons, the activation function is ReLU, and 40% of the network connections are randomly discarded; i. The second fully connected layer: contains 1024 neurons, the activation function is ReLU, and 40% of the network connections are randomly discarded; j. The third fully connected layer: contains 512 neurons, and the activation function is ReLU; k. Classification output layer: contains 7 output neurons and uses softmax as the activation function for sentiment classification.

5. The method for emotion analysis based on facial expression according to claim 4, characterized in that: In the emotion analysis and evaluation step, a machine learning or deep learning method is used to train an emotion classification model for matching and classifying the recognized facial expressions with emotion categories. The emotion classification model includes a support vector machine (SVM) or a decision tree or a neural network.

6. The method for emotion analysis based on facial expression according to claim 4 or 5, characterized in that: In the expression recognition and analysis step, the emotion classification model is trained using the YOLOv5 model for target detection. The model includes the following modules: a. Simplified convolutional layer: Contains convolution operation, batch normalization and ReLU activation function, convolution kernel size is 3, stride is 1, and automatic padding makes the input and output size the same; b. CSPNet module: It includes 1×1 convolution of the number of segmented channels, multi-layer 3×3 convolution and the final 1×1 convolution. It enhances the feature representation capability by splicing multiple intermediate results. c. YOLOv5 Backbone: contains multiple convolutional layers and CSP modules to extract features layer by layer. The beginning of the network contains a 3×3 convolutional layer, followed by multiple CSP modules, which increase the number of channels and feature map size layer by layer. d. YOLOv5 Head: Contains a 1×1 convolutional layer and a final detection layer that predicts the category and bounding box for each pixel.

7. The method for emotion analysis based on facial expression according to claim 5, characterized in that: The sentiment analysis and evaluation step also includes feature extraction and representation, and the method of feature extraction and representation includes converting the facial image into a feature vector that can be processed by the classification algorithm, and the features include local binary pattern (LBP), principal component analysis (PCA), and face embedding.

8. The method for emotion analysis based on facial expression according to claim 5, characterized in that: The optimization of the training of the emotion classification model includes collecting a large number of labeled facial expression data sets, using these data to train the emotion classification model, and optimizing the model through methods such as cross-validation to improve the generalization ability and accuracy of the model.

9. The method for emotion analysis based on facial expression according to claim 8, characterized in that: The training process of the sentiment classification model specifically includes: S1. Data enhancement: Perform data enhancement operations such as rotation, cropping, mirroring, and brightness adjustment on the training data to improve the robustness of the model; S2. Loss function: Use the cross entropy loss function to measure the gap between the model prediction results and the actual labels; S3. Optimization algorithm: Adopt adaptive learning rate optimization algorithm to update model parameters; S4. Learning rate adjustment: dynamically adjust the learning rate according to the change of loss during training, using the learning rate decay strategy; S5. Model evaluation: Model evaluation is performed through the validation set to monitor the performance changes of the model during the training process and prevent overfitting; The training of the sentiment classification model can also be performed by integrating multiple sub-models to perform sentiment analysis to improve classification accuracy and robustness. The specific steps include: a. Sub-model integration: training multiple models with different structures or parameters, and improving overall performance through integrated learning methods (Bagging or Boosting); b. Model fusion: The prediction results of multiple models are integrated by weighted average and voting methods to obtain the final sentiment classification result.

10. The method for emotion analysis based on facial expression according to any one of claims 1 to 9, characterized in that: The hardware architecture of the system for facial expression-based emotion analysis includes: High-performance image acquisition equipment: used to collect employee facial images; Computing resources: high-performance servers or cloud computing resources to run facial expression recognition and sentiment analysis algorithms; Storage device: used to store collected image data and analysis results; Network communication equipment: used to achieve efficient communication between data collection, transmission and processing; The software architecture of the system includes: Data collection module: responsible for collecting and preprocessing employees’ facial images; Expression recognition module: Use pre-trained deep learning models to perform expression recognition on facial images; Sentiment analysis module: sentiment classification and analysis based on expression recognition results; Result feedback module: Feedback the sentiment analysis results to the management system and generate corresponding reports and suggestions; System management module: used for system configuration, monitoring and maintenance.

Images