Complex expression recognition method based on convolutional neural network

Abstract

The invention relates to a complex expression recognition method based on a convolutional neural network, and aims to improve the expression recognition accuracy in social robots and customer service automation. According to the method, multiple modules of data preprocessing, feature extraction, model design, data enhancement, attention mechanism, multi-task learning, model optimization and the like are integrated, so that effective recognition of complex expressions expressed by mixed emotions is realized. By constructing a high-quality data set containing diversified expression samples, the method can robustly identify expressions under different environmental conditions, provides personalized interaction ability for a social robot by analyzing the emotional state of a user in real time, and improves the response quality and efficiency of customer service. Experiments prove that the method has remarkable advantages in the aspects of improving the recognition accuracy and improving the user experience.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to a complex expression recognition method based on convolutional neural networks. Background Art

[0002] Facial expression recognition technology has long been a key research area in the fields of artificial intelligence and computer vision. This technology analyzes and interprets facial expressions, identifying individual emotional states such as happiness, sadness, and anger. This technology has broad application prospects in a variety of fields, including human-computer interaction, security monitoring, and mental health assessment.

[0003] With the continuous advancement of technology, expression recognition technology has evolved from its initial rule-based approach to the use of machine learning algorithms, particularly deep learning algorithms. Deep learning algorithms, particularly convolutional neural networks (CNNs), have gained popularity due to their powerful capabilities in image processing tasks. CNNs can automatically learn complex and abstract features from images, thereby improving the accuracy of expression recognition.

[0004] Despite significant progress in expression recognition technology, recognizing complex expressions, particularly those expressing mixed emotions, remains challenging. Complex expressions often contain a mix of emotions, such as surprise and fear, or happiness and sadness. Recognizing these expressions requires models that can capture more subtle and complex facial changes. Furthermore, the recognition of complex expressions is affected by factors such as individual differences, cultural background, and environmental conditions. Existing expression recognition systems typically focus on recognizing basic emotions and have limited capabilities for recognizing complex expressions. These systems may be unable to accurately distinguish the nuances of mixed emotions, resulting in insufficient accuracy and reliability in recognition results.

[0005] Therefore, developing a deep learning model that can effectively recognize complex expressions is crucial to improving the performance of expression recognition technology in practical applications. Summary of the Invention

[0006] The purpose of this invention is to provide a complex expression recognition method based on convolutional neural networks. This method can accurately identify and classify complex expressions that express mixed emotions, and is particularly suitable for social robots and customer service automation systems. To achieve this goal, the present invention adopts the following technical solutions: In this invention, data preprocessing is the primary step in the complex expression recognition process, aiming to improve the efficiency of subsequent model training and recognition accuracy. First, advanced face detection techniques, such as a Haar feature cascade classifier or a deep learning-based face detection model, are used to scan the input image to precisely locate and crop the facial region. This step ensures that subsequent processing focuses on the face, eliminating irrelevant background interference. Next, the cropped facial image undergoes normalization, including resizing to match the model's input dimensions, converting to grayscale, and normalizing pixel values. These operations not only simplify the data but also enable the model to learn more quickly. Furthermore, histogram equalization enhances image contrast, effectively mitigating the impact of uneven lighting on expression recognition. A posture correction step is implemented. Using facial keypoint detection technology, specific keypoints on the face are detected and, based on these keypoints, the facial image is corrected to ensure a frontal or near-frontal posture. This helps the model more accurately capture the characteristics of the expression. De-noising is then performed on the image to reduce noise interference. By applying denoising algorithms such as Gaussian blurring or median filtering, the image is further purified, providing clearer image data for feature extraction. Histogram equalization is then performed to further enhance the expressive features in the image, providing high-quality input data for subsequent convolutional neural network model training. This series of carefully designed preprocessing steps lays a solid foundation for accurate recognition of complex expressions.

[0007] Feature extraction is a core step in expression recognition. Its purpose is to extract features that effectively represent facial expressions from preprocessed facial images. In this paper, a convolutional neural network (CNN) is used to perform this task. CNNs are widely used in image processing due to their unique advantages. The CNN model consists of multiple convolutional and pooling layers. The combination of these layers automatically learns local features from facial images and their spatial hierarchical structure. In each convolutional layer, a set of learnable filters (or convolution kernels) is applied. These filters slide across the image, extracting features by calculating the dot product between the filters and local regions. This process captures visual patterns such as edges, corners, and textures, which are critical information for expression recognition. Following each convolutional layer, an activation function, typically Rectified Linear Unit (ReLU), is introduced. The ReLU activation function introduces nonlinearity, enabling CNNs to learn and model more complex feature representations. Nonlinear activation functions are key to CNNs' ability to address nonlinear problems, allowing them to not only detect simple patterns but also capture more complex feature combinations. Pooling layers are used to reduce the spatial dimensionality of features, which not only reduces the computational effort of subsequent layers but also provides invariance to feature positions within the image. Max pooling is a commonly used pooling strategy that achieves dimensionality reduction and feature abstraction by selecting the maximum value in each local region. This helps the network become more robust to facial expression recognition, maintaining performance even when facial pose and position change.

[0008] Convolutional neural network (CNN) models are designed to process and recognize the diversity and complexity of complex facial expressions. The CNN model's architecture is optimized to improve its ability to recognize complex emotional expressions. Key technical steps in the model's design are outlined below: It begins with multiple convolutional layers, each equipped with a set of trainable filters (kernels). These kernels are set to 3x3 in size and perform convolution operations with a stride of 1 to capture local features in the image. Each convolutional layer is followed by a Reluctant Unit (ReLU) activation function, introducing nonlinearity and enabling the model to learn more complex feature representations. After each convolutional layer, a max pooling layer is added, operating with a 2x2 window and a stride of 2. This configuration reduces the spatial dimensionality of the feature map, reducing computational effort, and by extracting the most representative features in the image, enhances the model's robustness to changes in facial expressions. To further optimize the model's training process, a batch normalization layer is added between the convolutional and fully connected layers to accelerate model convergence and improve generalization. At the same time, a Dropout layer was introduced into the model, and the Dropout rate was set to 0.5 to randomly discard the activation outputs of some neurons, effectively reducing the risk of overfitting. After extracting features through multiple convolutional and pooling layers, the features are integrated through one or more fully connected layers. Finally, the model outputs the final expression classification result through a Softmax classification layer. The Softmax layer converts the output of the fully connected layer into a probability distribution, so that each output node represents the probability of an expression category. During the training of the model, the cross-entropy loss function was used to calculate the difference between the predicted expression and the real expression. Adam was selected as the optimizer, which combines the advantages of SGD and RMSprop and accelerates the convergence of the model through adaptive learning rate.

[0009] Data augmentation is a key technique for improving the generalization capabilities of deep learning models, particularly when handling complex expression recognition tasks. It helps the model learn more diverse expression features, thereby improving its robustness in practical applications. In this paper, a series of image processing operations are used for data augmentation. The specific technical steps are as follows: Each facial image in the training set is randomly rotated within a pre-set range, such as ±15 degrees. This rotation simulates the different angles people may use when communicating, enhancing the model's adaptability to changes in expression angles. To simulate facial images taken at different distances, the images are randomly scaled. The scaling ratio is within a certain range, such as 90% to 110%, to ensure the integrity of the image content while allowing the model to learn to handle facial images of different sizes. To increase data diversity, the images are horizontally flipped, i.e., flipped along the vertical axis. This operation simulates expressions from different perspectives, helping the model learn more comprehensive expression features. To address the issue of expression recognition under different lighting conditions, the images are randomly brightness-adjusted. By varying the image's brightness, we simulate different lighting environments, from dark to bright, improving the model's recognition capabilities under varying lighting conditions. In practice, these enhancements aren't performed in isolation but can be combined. For example, the same image can be rotated, scaled, and brightness adjusted to produce a wider range of variations.

[0010] The attention mechanism is used to enhance the performance of CNN models in complex expression recognition tasks. The introduction of the attention mechanism, particularly the gating mechanism, enables the model to more closely learn facial regions most relevant to emotional expression, such as the eyes, eyebrows, and mouth. The following are the specific technical steps and parameter settings for implementing the attention mechanism: First, attention weights are calculated for each pixel. This process typically involves a CNN-based encoder-decoder architecture. In the encoder, a convolutional layer is used to extract features. This convolutional layer is configured with 32 filters of size 3x3 and a stride of 1, which helps capture local features in the image. Next, a ReLU activation function introduces nonlinearity, followed by a 2x2 max pooling layer to reduce the spatial dimensionality of the features. After obtaining the initial attention weights, a gating mechanism is used to further adjust these weights. The gating mechanism typically includes a sigmoid function, which outputs a value between 0 and 1 indicating the importance of each pixel. For example, the output of the sigmoid function can be considered a mask that is used to weight the original feature map to highlight areas relevant to expression. The calculated attention weights are applied to the original feature map, and through element-by-element multiplication, important features are enhanced and minor features are suppressed. This step can be expressed using the following formula: `Attention_Refined_Features = Features * Attention_Weights`, where `Features` are the original features extracted by CNN and `Attention_Weights` are the calculated attention weights. The recalibrated features are used for expression classification. A Dropout layer is added before the fully connected layer, and the Dropout rate is set to 0.5 to reduce the risk of overfitting. Finally, the final expression classification result is output through a Softmax classification layer, and each output node represents the probability of an expression category.

[0011] Multi-task learning (MTL) is applied to CNN models to simultaneously learn tasks such as expression recognition, gender recognition, and age estimation. This multi-task learning approach helps the model learn richer and more robust feature representations. The following are the specific technical steps and parameter settings for implementing multi-task learning: The designed CNN model contains shared feature extraction layers, which are responsible for learning common features that are useful for all tasks. Above the shared layers, independent branch networks are designed for each task, which learn task-specific features. For example, for expression recognition, a network with three convolutional layers might be used, each followed by a ReLU activation function and a 2x2 max pooling layer to extract expression-related features. For gender recognition and age estimation, specific network branches are designed. These branches begin after the shared feature extraction layer and each branch contains additional convolutional and fully connected layers to meet the specific requirements of the respective task. For example, the gender recognition branch might contain a convolutional layer with 128 filters, while the age estimation branch might contain a convolutional layer with 256 filters, each followed by a ReLU activation function. A specific loss function is defined for each task, and these loss functions are weighted and summed to form the overall loss for multi-task learning. The weights are chosen based on the difficulty and relevance of the tasks. For example, the expression recognition task may be given a higher weight because it is potentially more challenging than the other tasks. The Adam optimizer is used to minimize the total loss, with a learning rate of 0.001 and momentum parameters β1 and β2 set to 0.9 and 0.999, respectively. In multi-task learning, some parameters are shared between tasks, while others are specific to each task. This strategy allows the model to transfer knowledge across tasks while maintaining specialization for each task. For example, convolution kernels in shared layers may be shared, while convolution kernels in task-specific branches are independent. During training, a batch size of 32 images is used, and each epoch consists of 100 iterations. Cross-validation is used to evaluate model performance, with each task evaluated on an independent test set. For example, the expression recognition task may use the OuluCASIA dataset, while the gender and age estimation tasks may use the Adience dataset.

[0012] Model optimization is achieved through carefully designed strategies designed to accelerate the learning process and improve model convergence. The following are the specific technical steps and parameter settings for model optimization: Stochastic Gradient Descent (SGD) with Nesterov momentum was chosen as the model optimizer. This approach accelerates SGD iterations by introducing momentum, enabling the model to converge to the optimal solution more quickly. In this implementation, the SGD optimizer's learning rate is initially set to 0.01, a parameter that controls the gradient descent step size during the initial stages of model training. The Nesterov momentum parameter (often denoted as `momentum`) is set to 0.9. This parameter controls the influence of historical gradients on current updates, helping the model maintain momentum in the relevant direction, thereby accelerating convergence and reducing oscillations. A learning rate monitor is used, specifically a scheduler based on the `ReduceLearningRate on Plateau` strategy. When the model's performance (e.g., validation set loss) does not improve significantly over a few consecutive epochs (the `patience` parameter, set to 10), the learning rate is proportionally reduced by a specified factor (the `factor` parameter, set to 0.1). This strategy helps to perform a more detailed search when the model is close to the optimal solution, avoiding falling into a local minimum too early. In addition, a minimum learning rate is set (`min_lr` parameter, set to 0.0001) to prevent the learning rate from dropping too low and affecting the model's learning ability. Closely monitor the performance of the model, especially the loss and accuracy on the validation set. Use the `ReduceLROnPlateau` callback function to dynamically adjust the learning rate. When the monitored indicators do not improve within the set patience period, the learning rate will be reduced by a preset factor. This adjustment strategy helps to adapt to the learning needs of the model at different stages of training, so that a larger learning rate is used for rapid descent in the early stage and a smaller learning rate is used for fine-tuning in the later stage.

[0013] Model training and evaluation are key steps in ensuring that CNN models can effectively recognize complex expressions. The CNN model was trained using a high-quality, carefully designed dataset containing a diverse range of expression samples. The following are the technical steps for model training and evaluation: The dataset contains a wide range of expression samples, covering not only the seven basic internationally recognized emotions, such as happiness, sadness, and anger, but also more complex mixed emotions. To improve the model's generalization, the dataset also includes samples from different genders, ages, ethnicities, and lighting conditions. This diversity ensures that the model can perform well in a variety of real-world situations. A CNN model was trained using this dataset. Training was iteratively performed with a batch size of 32 samples, with each epoch consisting of 100 iterations. This setting ensures that the model fully learns every expression sample in the dataset. An early stopping strategy was also implemented; training was automatically terminated if performance on the validation set did not improve significantly within five consecutive epochs to prevent overfitting. After model training, recognition accuracy was evaluated on an independent test set. The test set also contains diverse expression samples and does not overlap with the training and validation sets, ensuring the fairness and accuracy of the evaluation results. Metrics such as precision, recall, and F1 score were used to measure model performance. A real-time expression recognition test was conducted. In this test, the model was required to recognize expressions from facial images captured in real time. Different experimental environments were set up, including varying lighting conditions and different facial poses, to simulate real-world usage scenarios. The test results demonstrated that the model was capable of fast and accurate recognition under a variety of conditions.

[0014] Integrating a trained, complex expression recognition model into a social robot involves a series of technical steps to ensure the robot can analyze and respond to user expressions in real time. The following are the specific steps to achieve this goal: First, integrate the trained CNN model into the social robot's software framework. This typically involves exporting, converting, and loading the model to ensure it runs efficiently within the robot's computing environment. For example, the model may need to be exported from PyTorch or TensorFlow frameworks to the ONNX format for cross-platform use. The social robot captures the user's facial images in real time using a built-in or external camera. The model then analyzes the captured images in real time to identify the user's expressions. This step requires the model to have low latency and high accuracy to respond promptly to changes in the user's emotions. Based on the detected emotional state, the social robot can adjust its interaction strategy. For example, if the model identifies sadness in the user, the robot may choose a gentler and more comforting communication style. This adjustment can be implemented through pre-set rules or more complex decision tree algorithms. Social robots can provide personalized services based on the user's emotional state. For example, in an educational setting, if a student is identified as confused or frustrated, the robot can provide additional explanation or encouragement. In customer service scenarios, robots might adjust their response strategies based on the user's emotional state to improve customer satisfaction. Social robots can also learn from user feedback to optimize their expression recognition and interaction strategies. For example, if a user expresses dissatisfaction with a robot's response, the robot can adjust its strategy to better suit the user's needs. When applying expression recognition technology to social robots, privacy and ethical issues need to be considered. Ensure the security of user data and adhere to appropriate ethical guidelines when collecting and analyzing user expression data.

[0015] Through the above technical solution, the present invention can achieve high-precision recognition of complex expressions and apply this technology to social robots and customer service automation systems, thereby improving the ability of these systems to understand user emotions and provide personalized services.

[0016] The above technical solution can bring the following technical effects: 1. Improved accuracy in complex expression recognition: By using deep convolutional neural networks and an attention mechanism, this invention is able to capture the nuances of facial expressions, including those that represent mixed emotions. This enables the model to not only recognize basic single emotions but also accurately identify and distinguish mixed emotions, such as expressions of both surprise and fear, significantly improving the accuracy of complex expression recognition.

[0017] 2. Enhanced model generalization: Data augmentation allows the model to learn from a diverse set of training samples, encompassing varying lighting conditions, facial poses, and expressions. This enhanced dataset allows the model to maintain high recognition accuracy even when faced with novel expressions or environments, thereby enhancing its generalization capabilities.

[0018] 3. Real-time Expression Recognition: By optimizing the model structure and training process, the proposed model can efficiently process image data and achieve real-time expression recognition in social robots and customer service automation systems. This means that robots and automated systems can instantly respond to user emotions during interactions, providing users with a more natural and personalized experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of the complex expression recognition method of the present invention. DETAILED DESCRIPTION

[0020] The following is a combination of the embodiments of the present invention Figure 1 The technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] The purpose of this invention is to provide a complex facial expression recognition method based on a convolutional neural network (CNN). This method can accurately identify and classify complex expressions containing mixed emotions and is particularly suitable for social robots and automated customer service systems. To achieve this goal, the present invention adopts the following technical solutions: In the present invention, data preprocessing is a key step in the complex expression recognition process, aiming to improve the efficiency of subsequent model training and recognition accuracy. The specific data preprocessing process is as follows: Using a deep learning-based face detection model, such as a Haar feature cascade classifier or a deep learning-based face detection model, the input image is scanned to accurately locate and crop the facial region. The detected facial region is automatically cropped to form a new image file. This step ensures that subsequent processing focuses on the facial portion, eliminating interference from irrelevant background. The facial region identified in the face detection step is cropped to facilitate subsequent processing. The cropped facial image is resized to a uniform size, such as 48x48 or 64x64 pixels, to meet the input requirements of the CNN model. The color image is converted to a grayscale image to reduce data dimensionality and increase processing speed. Grayscaling is achieved by calculating a weighted sum in the RGB color space. Image pixel values are normalized from the [0, 255] range to the [0, 1] range, which helps the model converge faster. Use histogram equalization technology to enhance the contrast of the image and reduce the impact of uneven lighting on expression recognition. Histogram equalization is achieved by adjusting the histogram distribution of the image to make the brightness distribution of the image more uniform. Use facial key point detection technology, such as the face landmark detector in the dlib library, to detect specific key points of the face (such as the corners of the eyes, the tip of the nose, the corners of the mouth, etc.), and perform posture correction based on these key points to ensure that the face image is in a frontal or near-frontal posture. Apply image denoising algorithms, such as Gaussian blur or median filtering, to reduce noise in the image. Denoising helps improve image quality and reduce the impact of noise on subsequent feature extraction. For example, you can add Gaussian noise (taking mean = 0, sigma = 0.1 as an example) and salt and pepper noise (proportion = 0.1 as an example) Figure 1 The image under test is shown in the figure before and after denoising. As can be seen, the noise in the image is significantly reduced, demonstrating that the median filter is more effective in noise reduction (compared to the mean filter). Furthermore, while removing noise, the filter can well preserve edge sharpness and image detail. Histogram equalization is then performed on the denoised image to further enhance contrast and make the facial features more pronounced. Histogram equalization transforms an image with a known grayscale probability density distribution into a new image with a uniform grayscale probability density distribution. This increases the dynamic range of pixel grayscale values, thereby enhancing the overall contrast of the image.

[0022] Feature extraction is a core step in expression recognition technology, responsible for extracting key facial features from preprocessed facial images. In this paper, a deep convolutional neural network (CNN) is employed to achieve this goal. The CNN model is meticulously designed, comprising multiple convolutional and pooling layers. These layers automatically learn local features and spatial hierarchical structures within the image, providing a solid foundation for expression recognition. In the model, each convolutional layer is equipped with multiple convolutional kernels, typically sized 3x3 or 5x5. These kernels slide across the image with a stride of 1x1, capturing local features such as edges and textures. After each convolution operation, a ReLU activation function (f(x) = max(0, x)) is applied to introduce nonlinearity, enabling the model to learn and simulate more complex feature representations. This nonlinear transformation is key to CNN's ability to address nonlinearities, allowing the network to not only detect simple patterns but also capture more complex feature combinations. To reduce the spatial dimensionality of features and minimize the computational effort of subsequent layers, pooling layers are introduced between convolutional layers. The pooling layer typically uses a 2x2 max pooling operation, sliding across the feature map with a 2x2 window and a 2x2 stride, selecting the maximum value within each window to achieve dimensionality reduction and feature abstraction. This pooling strategy not only reduces the spatial size of the data but also provides the model with invariance to small positional changes, making it more robust for facial expression recognition. Furthermore, to further improve the model's generalization and avoid overfitting, a dropout layer is added to the CNN to randomly discard the activation outputs of a portion of the neurons, with a dropout rate typically set to 0.5. This regularization technique reduces overfitting of the model on the training data and improves its performance on unseen data.

[0023] Convolutional neural network (CNN) models are specialized for recognizing and classifying complex facial expressions. This model utilizes a carefully designed network architecture and specific parameter settings to enhance its ability to recognize complex emotional expressions. The following are the key technical steps and parameter settings for model design: Starting with multiple convolutional layers, each layer is equipped with a 3x3 kernel and a stride of 1. This configuration helps capture local features in the image, such as edges and textures. Each convolutional layer uses the ReLU activation function, with the formula `f(x) = max(0, x)`, to introduce nonlinearity, enabling the model to learn more complex feature representations. For example, the first layer may use 32 filters, and the second layer may use 64 filters. ReLU activation after each layer ensures the introduction of nonlinearity. After each convolutional layer, a max pooling layer is added with a pooling window of 2x2 and a stride of 2. This configuration helps reduce the spatial dimensionality of the feature map, reducing computational effort, and enhances the model's robustness to changes in facial expressions by extracting the most representative features in the image. To further optimize the model training process, a batch normalization layer is added between the convolutional and fully connected layers. This helps accelerate model convergence and improve generalization. A dropout layer with a dropout rate of 0.5 is also introduced into the model to randomly discard a portion of neuron activation outputs, effectively reducing the risk of overfitting. After feature extraction, the features are integrated through one or more fully connected layers. For example, the output of the last convolutional layer may be flattened and input into a fully connected layer with 128 neurons. Finally, the model outputs the final expression classification result through a softmax classification layer. The softmax layer converts the output of the fully connected layer into a probability distribution, so that each output node represents the probability of a certain expression category. During model training, the cross-entropy loss function is used to calculate the difference between the predicted expression and the ground-truth expression. Adam is used as the optimizer, which combines the advantages of SGD and RMSprop and accelerates model convergence through adaptive learning rate. The learning rate is initially set to 0.001 and is adjusted during training based on performance on the validation set.

[0024] Data augmentation techniques were used to improve the generalization capabilities of complex expression recognition models. Through a series of carefully designed image processing operations, diverse training samples were successfully generated from the original training set. The specific technical steps and parameter settings are as follows: A random rotation was applied to each face image. The rotation angle ranged from -15 degrees to +15 degrees, adjusted in steps of 0.5 degrees. This careful rotation setting not only increased the diversity of the dataset but also enabled the model to adapt to facial expressions at various angles. The images were randomly scaled. The scaling ratio ranged from 90% to 110% in 5% increments. This scaling strategy ensured the integrity of the image content while enabling the model to adapt to inputs of varying resolutions. The images were horizontally flipped. This flipping operation was performed with a 50% probability, meaning each image had a 50% chance of being flipped, improving the model's ability to recognize left and right facial expressions. The images were randomly brightness-adjusted. The brightness range was set to -20% to +20% in 5% increments. This brightness adjustment simulates varying lighting conditions, from dim to bright, enhancing the model's recognition capabilities under a variety of lighting conditions. In practice, these augmentation operations can be combined to generate more data variation. For example, the same image might first undergo a -10-degree rotation, then a 95% scale, and finally a 10% brightness boost. This combined augmentation strategy significantly enriches the training dataset, exposing the model to a wider range of expression variations and environmental conditions during training.

[0025] The attention mechanism is used to enhance the performance of convolutional neural networks (CNNs) in complex expression recognition tasks. The introduction of the attention mechanism, particularly the gating mechanism, enables the model to more closely learn facial regions most relevant to emotional expression, such as the eyes, eyebrows, and mouth. The following are the specific technical steps and parameter settings for implementing the attention mechanism: First, attention weights are calculated for each pixel. This process typically involves a CNN-based encoder-decoder architecture. In the encoder, a convolutional layer is used to extract features. This convolutional layer is configured with 32 filters of size 3x3 and a stride of 1, which helps capture local features in the image. Next, a ReLU activation function introduces nonlinearity, followed by a 2x2 max pooling layer to reduce the spatial dimensionality of the features. After obtaining the initial attention weights, a gating mechanism is used to further adjust these weights. The gating mechanism typically includes a sigmoid function, which outputs a value between 0 and 1 indicating the importance of each pixel. For example, the output of the sigmoid function can be considered a mask that is used to weight the original feature map to highlight areas relevant to expression. The calculated attention weights are applied to the original feature map, and through element-by-element multiplication, important features are enhanced and minor features are suppressed. This step can be expressed using the following formula: `Attention_Refined_Features = Features * Attention_Weights`, where `Features` are the original features extracted by CNN and `Attention_Weights` are the calculated attention weights. The recalibrated features are used for expression classification. A Dropout layer is added before the fully connected layer, and the Dropout rate is set to 0.5 to reduce the risk of overfitting. Finally, the final expression classification result is output through a Softmax classification layer, and each output node represents the probability of an expression category.

[0026] Multi-task learning (MTL) is applied to CNN models to simultaneously learn tasks such as expression recognition, gender recognition, and age estimation. This multi-task learning approach helps the model learn richer and more robust feature representations. The following are the specific technical steps and parameter settings for implementing multi-task learning: The designed CNN model contains shared feature extraction layers, which are responsible for learning common features that are useful for all tasks. Above the shared layers, independent branch networks are designed for each task, which learn task-specific features. For example, for expression recognition, a network with three convolutional layers might be used, each followed by a ReLU activation function and a 2x2 max pooling layer to extract expression-related features. For gender recognition and age estimation, specific network branches are designed. These branches begin after the shared feature extraction layer and each branch contains additional convolutional and fully connected layers to meet the specific requirements of the respective task. For example, the gender recognition branch might contain a convolutional layer with 128 filters, while the age estimation branch might contain a convolutional layer with 256 filters, each followed by a ReLU activation function. A specific loss function is defined for each task, and these loss functions are weighted and summed to form the overall loss for multi-task learning. The weights are chosen based on the difficulty and relevance of the tasks. For example, the expression recognition task may be given a higher weight because it is potentially more challenging than the other tasks. The Adam optimizer is used to minimize the total loss, with a learning rate of 0.001 and momentum parameters β1 and β2 set to 0.9 and 0.999, respectively. In multi-task learning, some parameters are shared between tasks, while others are specific to each task. This strategy allows the model to transfer knowledge across tasks while maintaining specialization for each task. For example, convolution kernels in shared layers may be shared, while convolution kernels in task-specific branches are independent. During training, a batch size of 32 images is used, and each epoch consists of 100 iterations. Cross-validation is used to evaluate model performance, with each task evaluated on an independent test set. For example, the expression recognition task may use the OuluCASIA dataset, while the gender and age estimation tasks may use the Adience dataset.

[0027] The core of the model optimization strategy is to use SGD (stochastic gradient descent) with Nesterov momentum to improve the learning efficiency and convergence speed of complex facial expression recognition models. Specifically, the SGD optimizer is used, with an initial learning rate set to 0.01. This value is a key parameter that controls the gradient descent step size during the initial training phase. The Nesterov momentum parameter, commonly referred to as `momentum`, is set to 0.9. This parameter helps the model maintain momentum in the relevant direction, accelerates convergence, and reduces oscillations during training. A learning rate monitor is introduced, based on the `Reduce Learning Rate on Plateau` strategy. When the model performance does not improve significantly for 10 consecutive epochs—the performance metric here can be loss or accuracy on the validation set—the learning rate is reduced proportionally by a pre-set factor. In this setting, the `patience` parameter is set to 10, and the `factor` parameter is set to 0.1, which means that the learning rate is reduced to 10% of its original value. Additionally, the `min_lr` parameter is set to 0.0001 to ensure that the learning rate does not drop too low, thereby affecting the model's learning ability. The `ReduceLROnPlateau` callback function is used to dynamically adjust the learning rate. This strategy allows the model to converge quickly using a larger learning rate early in the training process, while using a smaller learning rate for more refined adjustments later in the training process. When the monitored performance metric does not improve within the set patience period, the learning rate is reduced by a preset factor, allowing for a more detailed search as the model approaches the optimal solution, avoiding premature entrapment in local minima.

[0028] Model training and evaluation are critical steps in ensuring that CNN models can effectively recognize complex expressions. The CNN model was trained using a high-quality dataset containing a variety of expression samples. These samples not only cover the seven basic internationally recognized emotions, such as happiness, sadness, and anger, but also include more complex mixed emotions. The dataset further includes expressions from individuals of different genders, ages, and ethnicities under various lighting conditions to improve the model's generalization ability. The sample size in the dataset was set to at least 10,000 images, ensuring diversity and comprehensiveness in model training. Iterative training was performed using a batch size of 32 samples, meaning that information from 32 images was considered in each update of the model weights. Each epoch consisted of 100 iterations, meaning the entire dataset was passed through 100 times. An early stopping strategy was also implemented, automatically terminating training if performance on the validation set did not improve significantly within five consecutive epochs to prevent model overfitting. The initial value of the learning rate is set to 0.001 and is dynamically adjusted according to the `Reduce LearningRate on Plateau` strategy. The `patience` parameter of this strategy is set to 10 and the `factor` parameter is set to 0.1, ensuring that the learning rate can be reduced when the model performance stops improving. During the performance evaluation phase, the recognition accuracy of the model was evaluated on an independent test set. The test set contains at least 2,000 images that were not used in the training process. These images also cover a variety of emotional expressions and environmental conditions. Indicators such as precision, recall, and F1 score are used to measure the performance of the model. The precision reached more than 90%, the recall rate exceeded 85%, and the F1 score exceeded 0.9. These indicators show the high recognition ability of the model.

[0029] In this invention, a carefully trained convolutional neural network (CNN) model is integrated into a social robot, enabling it to analyze user expressions in real time and respond accordingly. This process involves multiple technical steps to ensure the robot can accurately capture and respond to changes in user emotions. The CNN model is exported to the ONNX format and optimized on the social robot's hardware to ensure efficient operation. The model's input size is set to 64x64 pixels, a reasonable balance between preserving expression features and computational efficiency. The model's inference time is kept within 200 milliseconds, ensuring the feasibility of real-time expression analysis. The social robot captures user facial images in real time using a 1080p resolution camera. The images are processed by the model, with a frame rate set to 5 frames per second to balance real-time performance and computational load. The CNN model uses the ReLU activation function and a learning rate of 0.001. These parameters are optimized during training to achieve optimal performance. Once a user's expression, such as happiness, sadness, or anger, is recognized, the robot adjusts its interaction strategy based on pre-set emotional response rules. For example, if a robot recognizes a user smiling, it might increase positive feedback in the conversation to enhance the user's positive experience. Social robots provide personalized services based on the user's emotional state. For example, in an educational setting, if the model identifies a student's expression of confusion, the robot might automatically adjust the difficulty of the instructional content. Furthermore, the robot records user feedback on its responses and uses this data to fine-tune its decision-making algorithms for even more accurate emotion recognition and response. When processing user expression data, ensure that all data is encrypted and that privacy regulations such as GDPR are followed during data collection and analysis. The robot's interactive interface clearly informs users of the manner and scope of data use to ensure informed consent.

[0030] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A complex expression recognition method based on convolutional neural network, characterized in that: The method comprises the following steps: s1. Data preprocessing: face detection, normalization, lighting adjustment, and posture correction of input images; s2. Feature extraction: using a convolutional neural network to extract facial features from pre-processed facial images; s3. Model design: building a CNN model consisting of multiple convolutional layers, pooling layers, fully connected layers, and a softmax classification layer; s4. Data augmentation: generating more training samples through image processing operations to enhance the generalization ability of the model; s5. The introduction of the attention mechanism enables the model to focus more on key facial areas that express emotions; s6. Multi-task learning: Simultaneously learning expression recognition and other related tasks to improve the model's ability to recognize expressions; s7. Model optimization, using the SGD method based on Nesterov momentum and a learning rate monitor to optimize the model and prevent overfitting.

2. The method according to claim 1, characterized in that The data preprocessing step also includes noise removal and histogram equalization processing.

3. The method according to claim 1, characterized in that In the feature extraction step, the convolutional neural network includes at least two convolutional layers and two pooling layers.

4. The method according to claim 1, wherein In the model design step, the CNN model also includes a batch normalization layer and a dropout layer.

5. The method according to claim 1, wherein In the data enhancement step, image processing operations include random rotation, scaling, flipping and brightness adjustment.

6. The method according to claim 1, characterized in that The attention mechanism is introduced into the step, and a gating mechanism is used to weight important expression features.

7. The method according to claim 1, characterized in that In the multi-task learning step, expression recognition and other tasks performed simultaneously include gender recognition and age estimation.

8. The method according to claim 1, wherein In the model optimization step, the learning rate monitor is based on the Reduce Learning Rate on Plateau strategy.

9. A computing device using the method according to any one of claims 1 to 8, characterized in that: The apparatus includes a processor, a memory, and an input / output device, wherein the memory stores a computer program for executing the method.

10. A computer-readable storage medium, characterized in that The medium stores a computer program, which is designed to execute the steps of the method according to any one of claims 1 to 8.

Images