Image processing method and device, storage medium and electronic device
By using a RedNet feature extractor composed of involution operators and a Transformer encoder, combined with attention mechanisms and Bayesian estimation, the problem of insufficient utilization of image details in the ViT model in emotion recognition is solved, and the accuracy of emotion classification is improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NEW ORIENTAL EDUCATION & TECH GRP CO LTD
- Filing Date
- 2021-12-02
- Publication Date
- 2026-07-10
Smart Images
Figure CN116229530B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing, and more specifically, to an image processing method, apparatus, storage medium, and electronic device. Background Technology
[0002] Emotion recognition is an unavoidable part of any interpersonal communication. People observe changes in the emotions of others to confirm whether their own behavior is reasonable and effective. With the continuous advancement of technology, emotion recognition can use different features for detection and identification, such as faces, voices, electroencephalograms (EEGs), and even speech content. Among these features, facial expressions are usually the easiest to observe.
[0003] In recent years, with the application of deep learning, especially the emergence of the ViT (Vision Transformer) model, the monopoly of networks based on convolution and pooling in classification tasks has been successfully broken. However, the underlying convolutional part of the ViT model is too simple, and the underlying network does not make good use of more detailed image information. Furthermore, there is no feature map size reduction transformation in the intermediate processing stage. Summary of the Invention
[0004] To address the problems existing in related technologies, this disclosure provides an image processing method, apparatus, storage medium, and electronic device.
[0005] To achieve the above objectives, the first aspect of this disclosure provides an image processing method, the method comprising:
[0006] Acquire a target image including facial information;
[0007] The target image is input into a pre-trained emotion classification network to obtain the emotion information represented by the facial information in the target image;
[0008] The emotion classification network includes a RedNet feature extractor composed of involution operators. The RedNet feature extractor is used to obtain a feature image based on the target image, and then obtain the emotion information based on the feature image.
[0009] Optionally, obtaining the emotion information based on the feature image includes:
[0010] The feature image is input into the Transformer encoder to obtain the feature vector corresponding to the target image. The Transformer encoder includes a multi-head self-attention module, a multilayer perceptron, and a layer normalization module.
[0011] The feature vector is input into a fully connected layer to obtain the emotional information represented by facial information in the target image.
[0012] Optionally, the training of the emotion classification network includes:
[0013] Obtain a training set, which includes multiple training images, each of which includes facial information and pre-labeled emotion tags corresponding to the facial information.
[0014] For any target training image in the training set, the target training image is input into the RedNet feature extractor in the initial emotion classification network to obtain the feature image of the target training image;
[0015] The feature image of the target training image is input into the Transformer encoder to obtain the feature vector corresponding to the target training image;
[0016] The feature vector corresponding to the target training image is input into a fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image.
[0017] Based on the predicted labels and the pre-labeled emotion labels on the target training images, the parameters in the emotion classification network are adjusted to obtain the trained emotion classification network.
[0018] Optionally, the fully connected layer includes an attention factor, and the step of inputting the feature vector corresponding to the target training image into the fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image includes:
[0019] The feature vector corresponding to the target training image is input into a fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image, as well as the weight information of the target training image.
[0020] The step of adjusting the parameters in the emotion classification network based on the predicted labels and the pre-labeled emotion labels of the training images includes:
[0021] Based on the predicted labels, the pre-labeled emotion labels on the target training images, and the weight information of the target training images, the parameters in the emotion classification network are adjusted using the cross-entropy loss function and regularization loss.
[0022] Optionally, the method further includes:
[0023] Obtain a test set, which includes multiple test images, each of which includes facial information and an emotion label pre-annotated to the facial information;
[0024] For any target test image in the test set, the target test image is input into the RedNet feature extractor in the trained emotion classification network to obtain the feature image of the target test image;
[0025] The feature image of the target test image is input into the Transformer encoder to obtain the feature vector corresponding to the target test image;
[0026] The feature vector corresponding to the target test image is input into the MC-dropout layer to determine the uncertainty information of the target test image;
[0027] Determine whether the uncertainty information of the multiple test images meets a preset rule. If the preset rule is met, the trained emotion classification network is used as the trained emotion classification network.
[0028] A second aspect of this disclosure provides an image processing apparatus, the apparatus comprising:
[0029] The acquisition module is used to acquire target images including facial information;
[0030] The emotion determination module is used to input the target image into a pre-trained emotion classification network to obtain emotion information represented by facial information in the target image;
[0031] The emotion classification network includes a RedNet feature extractor composed of involution operators. The RedNet feature extractor is used to obtain a feature image based on the target image, and then obtain the emotion information based on the feature image.
[0032] Optionally, the emotion determination module is specifically used for:
[0033] The feature image is input into the Transformer encoder to obtain the feature vector corresponding to the target image. The Transformer encoder includes a multi-head self-attention module, a multilayer perceptron, and a layer normalization module.
[0034] The feature vector is input into a fully connected layer to obtain the emotional information represented by facial information in the target image.
[0035] Optionally, the device includes:
[0036] The second acquisition module is used to acquire a training set, which includes multiple training images. Each training image includes facial information and an emotion label pre-annotated to the facial information.
[0037] The feature extraction module, for any target training image in the training set, inputs the target training image into the RedNet feature extractor in the initial emotion classification network to obtain the feature image of the target training image;
[0038] The feature vector determination module is used to input the feature image of the target training image into the Transformer encoder to obtain the feature vector corresponding to the target training image;
[0039] The prediction module is used to input the feature vector corresponding to the target training image into the fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image;
[0040] The adjustment module is used to adjust the parameters in the emotion classification network according to the predicted labels and the emotion labels pre-annotated in the target training image, so as to obtain the trained emotion classification network.
[0041] A third aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the first aspects of this disclosure.
[0042] A fourth aspect of this disclosure provides an electronic device, comprising:
[0043] A memory on which computer programs are stored;
[0044] A processor for executing the computer program in the memory to implement the steps of the method according to any one of the first aspects of this disclosure.
[0045] By using the above technical solution, the RedNet structure composed of involution operators is used as a feature extractor to perform preliminary processing on the image input to the emotion classification network, extract local details of the image, and input the obtained feature image into the downstream module of the emotion classification network, which effectively improves the final accuracy of the emotion information output by the emotion classification network.
[0046] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description
[0047] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:
[0048] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment;
[0049] Figure 2This is a schematic diagram illustrating a sentiment classification network during the training phase according to an exemplary embodiment;
[0050] Figure 3 This is a schematic diagram illustrating an emotion classification network during a testing phase, according to an exemplary embodiment.
[0051] Figure 4 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment;
[0052] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment;
[0053] Figure 6 This is another block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0054] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0055] Emotion recognition is an unavoidable part of any interpersonal communication. People observe changes in the emotions of others to confirm whether their own behavior is reasonable and effective. With the continuous advancement of technology, emotion recognition can use different features for detection and identification, such as faces, voices, electroencephalograms (EEGs), and even speech content. Among these features, facial expressions are usually the easiest to observe.
[0056] Generally, facial expression recognition systems consist of three main stages: face detection, feature extraction, and expression recognition. In the face detection stage, multiple face detectors, such as MTCNN and RetinaFace networks, are used to locate faces in complex scenes, and detected faces can be further aligned. For feature extraction, past research has proposed various methods to capture facial geometric and appearance features induced by facial expressions. Based on feature type, these can be categorized into engineered features and learning-based features. Engineered features can be further divided into texture-based features, geometry-based global features, etc.
[0057] In recent years, with the application of deep learning, especially the emergence of the ViT (Vision Transformer) model, the monopoly of networks based on convolution and pooling in classification tasks has been successfully broken. However, the underlying convolutional part of the ViT model is too simple, and the underlying network does not make good use of more detailed image information. Furthermore, there is no feature map size reduction transformation in the intermediate processing stage.
[0058] Figure 1This is a flowchart illustrating an image processing method according to an exemplary embodiment. The execution subject of this method can be a terminal such as a mobile phone, computer, or laptop, or a server, such as... Figure 1 As shown, the method includes:
[0059] S101. Obtain the target image including facial information.
[0060] The facial information in the target image can include the facial information of only one person, or it can include the facial information of multiple people.
[0061] S102. Input the target image into a pre-trained emotion classification network to obtain the emotion information represented by the facial information in the target image.
[0062] It is understandable that this emotional information can represent the probability values of emotions such as happiness, sadness, crying, and laughing corresponding to the facial information of the task in the target image.
[0063] The emotion classification network includes a RedNet feature extractor composed of involution operators. The RedNet feature extractor is used to obtain a feature image based on the target image, and then obtain the emotion information based on the feature image.
[0064] Those skilled in the art should understand that in the traditional ViT model, the image is cut into uniform segments with equal steps. However, this may cause the loss or misalignment of some features in the segmentation of local information. Image processing is different from natural language processing tasks where text has contextual relationships, and the relationship between pixels has a larger granularity of continuity.
[0065] Furthermore, the involution operator possesses channel invariance and space specificity. Its design is the opposite of convolution, sharing the kernel in the channel dimension while employing a space-specific kernel for more flexible modeling. Compared to convolution's sharing of spatial weights, the involution kernel pays different attention to different spatial locations, allowing for more effective extraction of diverse target features. Moreover, without increasing parameter computation, it shares and transfers feature weights across different spatial locations, precisely aligning with the space-specific design principle. This design, from convolution to involution, reallocates computational resources, optimizing performance by allocating limited resources to their most efficient positions. Therefore, we use RedNet, composed of involution operators, as our feature extractor, achieving better results than ResNet with a smaller parameter count.
[0066] In this embodiment, a RedNet structure composed of involution operators is used as a feature extractor to perform preliminary processing on the image input to the emotion classification network, extract local details of the image, and input the obtained feature image into the downstream module of the emotion classification network, which effectively improves the final accuracy of the emotion information output by the emotion classification network.
[0067] In some optional embodiments, obtaining the emotion information based on the feature image includes:
[0068] The feature image is input into the Transformer encoder to obtain the feature vector corresponding to the target image. The Transformer encoder includes a multi-head self-attention module, a multilayer perceptron, and a layer normalization module.
[0069] The feature vector is input into a fully connected layer to obtain the emotional information represented by facial information in the target image.
[0070] It is understood that the feature image may include multiple feature sub-image patches, and inputting the feature image into the Transformer encoder includes: stretching the multiple feature sub-image patches and then inputting them into the Transformer encoder respectively.
[0071] The multi-head self-attention (MSA) module linearly connects multiple attention outputs to the desired dimension. Multiple attention heads can be used to understand local and global dependencies in an image. The multi-layer perceptron (MLP) contains two layers of Gaussian Error Linear Units (GELUs) and layer normalization (LN), which can be used to improve training time and generalization performance. Residual connections are applied after each patch because they allow gradients to flow directly through the network without passing through nonlinear layers.
[0072] Those skilled in the art will understand that Convolutional Neural Networks (CNNs), when applied to the facial recognition domain, can be trained on a dataset to extract and learn a facial expression recognition system containing key features. However, it is noteworthy that, regarding facial expressions, many cues originate from certain facial features, such as the mouth and eyes, while other parts, such as the background and hair, play a relatively minor role in the output. This means that, ideally, the model framework should focus only on the important facial features, paying less attention to those sensitive to other facial regions, and exhibiting good generalization ability to special cases such as occlusion and blurring. In this work, we propose a Transformer-based framework for facial expression recognition that considers the above observations and utilizes an attention mechanism to focus on prominent facial features. Using Transformer encoding, instead of a deep convolutional model, achieves very high accuracy.
[0073] By employing the above scheme and utilizing the Transformer encoder, an attention mechanism is used to focus on prominent facial features, ensuring the accuracy of the emotional information output by the emotion classification network.
[0074] In some alternative embodiments, training the emotion classification network includes:
[0075] Obtain a training set, which includes multiple training images, each of which includes facial information and pre-labeled emotion tags corresponding to the facial information.
[0076] For any target training image in the training set, the target training image is input into the RedNet feature extractor in the initial emotion classification network to obtain the feature image of the target training image;
[0077] The feature image of the target training image is input into the Transformer encoder to obtain the feature vector corresponding to the target training image;
[0078] The feature vector corresponding to the target training image is input into a fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image.
[0079] Based on the predicted labels and the pre-labeled emotion labels on the target training images, the parameters in the emotion classification network are adjusted to obtain the trained emotion classification network.
[0080] Using the above scheme, an untrained initial emotion classification network is trained based on a training set of multiple training images, including facial information and emotion labels pre-annotated to the corresponding facial information, to obtain an emotion classification network that can accurately identify and classify the emotions represented by facial information in the images.
[0081] In some alternative embodiments, the fully connected layer includes an attention factor, and the step of inputting the feature vector corresponding to the target training image into the fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image includes:
[0082] The feature vector corresponding to the target training image is input into a fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image, as well as the weight information of the target training image.
[0083] The step of adjusting the parameters in the emotion classification network based on the predicted labels and the pre-labeled emotion labels of the training images includes:
[0084] Based on the predicted labels, the pre-labeled emotion labels on the target training images, and the weight information of the target training images, the parameters in the emotion classification network are adjusted using the cross-entropy loss function and regularization loss.
[0085] The above scheme uses an attention factor added to the fully connected layer to determine the true accuracy of the samples in the training set. A high value indicates good sample performance, high accuracy, and a greater "role" played during training; conversely, a low value indicates poor sample performance, low accuracy, and unsatisfactory training. Through this factor, the neural network focuses its attention on samples that are actually more effective and perform better, thus effectively improving training accuracy.
[0086] In some other embodiments, training the emotion classification network further includes feeding the training set into a Self-Cure Network (SCN) to automatically correct erroneous labels in the samples. The SCN includes a self-attention importance weighting module and a relabeling module.
[0087] The self-attention importance weighting module is used for each sample x in the training set. i Generate a weight α i , as a sample x in the training set i The importance of the self-attention weighted module is measured. The self-attention importance weighting module is trained using RR-loss (Rank Regularization loss).
[0088] The specific steps for calculating RR-loss include: calculating a batch of samples according to α... i Sort the samples and divide them into two groups according to a ratio β: high score and low score. The high score group has β*N = M samples, and the low score group has NM samples. Then: LRR =max{0,δ1-(α)} H -α L )},
[0089] Among them, L RR Represents RR-loss, α H α represents the average weight of the high-weight group. L Let α represent the average weight of the low-weight group, and α H and α L Satisfy the following formula:
[0090] Understandably, δ1 is a fixed or learnable value used to separate the weighted mean of the high and low groups.
[0091] In other embodiments, dividing the samples into high-scoring and low-scoring groups according to a ratio β includes:
[0092] In the distance formula argmax M (min i∈[0,M) α i -max i∈[M,N) α i If the condition is not met, the ratio β is manually calibrated. Within this range, the above distance formula is used for grouping.
[0093] Understandably, using a fixed hyperparameter β to group training samples is equivalent to making an assumption about the proportion of mislabeled samples in the data. However, in reality, we often don't know the distribution of mislabeled samples in the data. On the other hand, even if we know the proportion of mislabeled samples in the overall data, the randomness of sampling will cause the proportion to vary from batch to batch, and using a fixed proportion will introduce some bias.
[0094] Once the self-attention importance weighting module has learned how to distinguish between high and low groups, the best grouping method should satisfy: argmax M distance(Ω H ,Ω L )
[0095] Among them, Ω H Ω represents the set of samples with high group weights. L This represents the set of samples in the low-weight group. Considering the ordered nature of these weights, the distance can be calculated using the formula argmax. M (min i∈[0,M) α i -max i∈[M,N) α i ).
[0096] By adopting this scheme, the samples in each batch are grouped according to their actual weights, which can achieve adaptive grouping while avoiding training instability.
[0097] Furthermore, considering the varying complexity of training set samples across different categories, the metrics used to assess the importance of each sample when calculating the confidence level for belonging to each category are not entirely consistent. Therefore, we extended α... i The dimension is changed from a scalar to a vector with a 1×c output category dimension. α is used when calculating the RR-loss. i The mean is constrained.
[0098] Using the above scheme, an adaptive grouping method is proposed, which groups samples according to their actual weights in each batch, effectively improving the accuracy of the weights output by the model.
[0099] In some alternative embodiments, the method further includes:
[0100] Obtain a test set, which includes multiple test images, each of which includes facial information and an emotion label pre-annotated to the facial information;
[0101] For any target test image in the test set, the target test image is input into the RedNet feature extractor in the trained emotion classification network to obtain the feature image of the target test image;
[0102] The feature image of the target test image is input into the Transformer encoder to obtain the feature vector corresponding to the target test image;
[0103] The feature vector corresponding to the target test image is input into the MC-dropout layer to determine the uncertainty information of the target test image;
[0104] Determine whether the uncertainty information of the multiple test images meets a preset rule. If the preset rule is met, the trained emotion classification network is used as the trained emotion classification network.
[0105] Those skilled in the art will understand that in related technologies, CNN models, attention models, and Transformer models are mathematically maximum likelihood estimation models. Maximum likelihood estimation models are unbiased and have fixed weights. However, in the real world, the weights of any model should tend towards a Gaussian distribution, not a fixed one. Therefore, maximum likelihood estimation cannot effectively estimate the uncertainty of data. Human expressions are inherently complex; for example, fear and surprise, or laughing until tears stream down one's face, are mixtures of different expressions, not a single expression. Therefore, using a model with fixed weights to estimate an uncertain task is inherently contradictory.
[0106] Those skilled in the art will understand that MC-dropout is a way of understanding dropout based on Bayesian theory, interpreting dropout as a Bayesian approximation of a Gaussian process. This allows ordinary models to possess the ability to evaluate uncertainty, much like Bayesian neural networks.
[0107] Specifically, using the MC-dropout layer only requires one input to be tested n times during testing to obtain a set of sampling points, thereby calculating the mean and variance. The variance can be used to evaluate the uncertainty of the prediction of samples in the test set. The larger the variance, the higher the uncertainty of the prediction.
[0108] In some implementations, during testing, the backbone outputs feature O b ∈R 1×p Typically, O b Will be related to the weights of the fully connected layer Multiplying them together, the formula is O fc =O b ·W fc , among them It will be used for further classification.
[0109] In other possible implementations, for W fc n samplings were performed. The weights obtained from the sampling are denoted as... The MC-dropout layer can then be defined using the following formula:
[0110] in An additional sampling dimension has been added. This is relative to O. fc , This is equivalent to the result of n samplings using dropout. The final classification result is obtained by calculating the mean value using the following formula:
[0111]
[0112]
[0113] Among them, softmax m The () function represents the function in Perform a softmax operation on the m-th dimension. n () indicates that in The average value is calculated along the n-dimensional vector. `max()` represents finding the maximum value of the vector. The uncertainty of the sample is calculated as follows:
[0114] Among them, variance n The function () represents the expression in The variance is calculated over the n-dimensional space. Function representation O mean The corresponding sample variance. This can be based on O. var The maximum value of the variance measures the uncertainty of the prediction result. The larger the variance, the higher the uncertainty.
[0115] Alternatively, dropout can be implemented in other layers, as long as the computation before that layer is run only once, and then the MC-dropout layer becomes a matrix operation.
[0116] Using this approach, during the testing phase, uncertainty analysis can be performed using Bayesian estimation by replacing the fully connected layer with an MC-dropout layer.
[0117] To enable those skilled in the art to better understand the technical solutions provided in this disclosure, this disclosure provides, as follows: Figure 2 The diagram shown illustrates a training phase of an emotion classification network 20 according to an exemplary embodiment. Figure 2 As shown, the emotion classification network 20 includes an input module 21, a RedNet feature extractor 22, a Transformer encoder 23, a fully connected layer 24, and a classifier 25 connected in series.
[0118] based on Figure 2The emotion classification network 20 shown is trained by: inputting the training set into the RedNet feature extractor 22 of the emotion classification network 20 through the input module 21 to obtain multiple feature images pctch for any training image in the training set; inputting the multiple feature images pctch into the Transformer encoder 23 to obtain the feature vector of any training image in the training set; inputting the feature vector into the fully connected layer 24 to obtain the probability value of each emotion category represented by facial information in the target image; inputting the probability value of each emotion category into the classifier 25 to obtain the emotion category with the highest probability; and adjusting the parameters of the emotion classification network 20 based on the emotion category and the pre-labeled information in the training set, using the cross-entropy loss function and regularization loss, to obtain the trained emotion classification network.
[0119] Furthermore, this disclosure also provides, for example Figure 3 The diagram shown illustrates an emotion classification network during a testing phase, according to an exemplary embodiment. Figure 3 As shown, the emotion classification network 30 includes a trained input module 31, a RedNet feature extractor 32, a Transformer encoder 33, an MC-dropout layer 34, and a classifier 35.
[0120] based on Figure 3 The emotion classification network 30 shown is tested as follows: The test set is input into the RedNet feature extractor 32 of the emotion classification network 30 via the input module 31 to obtain multiple feature images pctch for any training image in the training set; these multiple feature images pctch are input into the Transformer encoder 33 to obtain a feature vector for any training image in the training set; the feature vector is input into the MC-dropout layer 34 for multiple samplings to obtain the probability value of each emotion category represented by facial information in the target image output by the MC-dropout layer 34 at each sampling; the probability values of each emotion category are input into the classifier 35 to obtain the emotion category with the highest probability; based on the emotion category and the pre-labeled information of the test set, it is determined whether the emotion classification network 30 meets the preset requirements.
[0121] based on Figure 3 as well as Figure 4 This emotion classification network architecture, based on SCN, is the first to combine RedNet and Transformer as feature extractors. It also utilizes RedNet and Bayesian-based MC-dropout. Furthermore, to handle blurred images and labels in the training set, it leverages and further improves upon the training methods found in SCN.
[0122] Figure 4 This is a block diagram illustrating an image processing apparatus 40 according to an exemplary embodiment. The apparatus 40 can be part of a terminal such as a mobile phone, or it can be part of a server. The apparatus 40 includes:
[0123] The first acquisition module 41 is used to acquire a target image including facial information;
[0124] The emotion determination module 42 is used to input the target image into a pre-trained emotion classification network to obtain emotion information represented by facial information in the target image;
[0125] The emotion classification network includes a RedNet feature extractor composed of involution operators. The RedNet feature extractor is used to obtain a feature image based on the target image, and then obtain the emotion information based on the feature image.
[0126] Optionally, the emotion determination module 42 is specifically used for:
[0127] The feature image is input into the Transformer encoder to obtain the feature vector corresponding to the target image. The Transformer encoder includes a multi-head self-attention module, a multilayer perceptron, and a layer normalization module.
[0128] The feature vector is input into a fully connected layer to obtain the emotional information represented by facial information in the target image.
[0129] Optionally, the device 40 further includes:
[0130] The second acquisition module is used to acquire a training set, which includes multiple training images. Each training image includes facial information and an emotion label pre-annotated to the facial information.
[0131] The first feature extraction module inputs any target training image in the training set into the RedNet feature extractor in the initial emotion classification network to obtain the feature image of the target training image.
[0132] The first feature vector determination module is used to input the feature image of the target training image into the Transformer encoder to obtain the feature vector corresponding to the target training image;
[0133] The prediction module is used to input the feature vector corresponding to the target training image into the fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image;
[0134] The adjustment module is used to adjust the parameters in the emotion classification network according to the predicted labels and the emotion labels pre-annotated in the target training image, so as to obtain the trained emotion classification network.
[0135] Optionally, the fully connected layer includes an attention factor, and the prediction module is specifically used for:
[0136] The feature vector corresponding to the target training image is input into a fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image, as well as the weight information of the target training image.
[0137] The adjustment module is specifically used for:
[0138] Based on the predicted labels, the pre-labeled emotion labels on the target training images, and the weight information of the target training images, the parameters in the emotion classification network are adjusted using the cross-entropy loss function and regularization loss.
[0139] Optionally, the device 40 further includes:
[0140] The third acquisition module is used to acquire a test set, which includes multiple test images. Each test image includes facial information and an emotion label pre-annotated to the facial information.
[0141] The second feature extraction module is used to input any target test image in the test set into the RedNet feature extractor in the trained emotion classification network to obtain the feature image of the target test image.
[0142] The second feature vector determination module inputs the feature image of the target test image into the Transformer encoder to obtain the feature vector corresponding to the target test image.
[0143] The first determining module is used to input the feature vector corresponding to the target test image into the MC-dropout layer to determine the uncertainty information of the target test image;
[0144] The second determining module is used to determine whether the uncertainty information of the multiple test images meets a preset rule. If the preset rule is met, the trained emotion classification network is used as the trained emotion classification network.
[0145] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0146] Figure 5This is a block diagram illustrating an electronic device 500 according to an exemplary embodiment. For example... Figure 5 As shown, the electronic device 500 may include a processor 501 and a memory 502. The electronic device 500 may also include one or more of a multimedia component 503, an input / output (I / O) interface 504, and a communication component 505.
[0147] The processor 501 controls the overall operation of the electronic device 500 to complete all or part of the steps in the image processing method described above. The memory 502 stores various types of data to support the operation of the electronic device 500. This data may include, for example, instructions for any application or method operating on the electronic device 500, and application-related data, such as images in training sets, test sets, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 503 may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 502 or transmitted via communication component 505. The audio component also includes at least one speaker for outputting audio signals. I / O interface 504 provides an interface between processor 501 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 505 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.
[0148] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method described above.
[0149] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the image processing method described above. For example, the computer-readable storage medium may be the memory 502 including program instructions described above, which may be executed by the processor 501 of the electronic device 500 to complete the image processing method described above.
[0150] Figure 6 This is a block diagram illustrating an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be provided as a server. (Refer to...) Figure 6 The electronic device 600 includes a processor 622, which may be one or more, and a memory 632 for storing computer programs executable by the processor 622. The computer programs stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 622 may be configured to execute the computer program to perform the image processing method described above.
[0151] Additionally, the electronic device 600 may also include a power supply component 626 and a communication component 650. The power supply component 626 can be configured to perform power management of the electronic device 600, and the communication component 650 can be configured to enable communication of the electronic device 600, such as wired or wireless communication. Furthermore, the electronic device 600 may also include an input / output (I / O) interface 658. The electronic device 600 can operate on an operating system, such as Windows Server, stored in memory 632. TM Mac OSX TM Unix TM Linux TM etc.
[0152] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the image processing method described above. For example, the non-transitory computer-readable storage medium may be the memory 632 including the program instructions described above, which may be executed by the processor 622 of the electronic device 600 to complete the image processing method described above.
[0153] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the image processing method described above when executed by the programmable device.
[0154] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.
[0155] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.
[0156] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.
Claims
1. An image processing method, characterized in that, The method includes: Acquire a target image including facial information; The target image is input into a pre-trained emotion classification network to obtain the emotion information represented by facial information in the target image; The emotion classification network includes a RedNet feature extractor composed of involution operators. The RedNet feature extractor is used to obtain a feature image based on the target image, and to obtain the emotion information based on the feature image. The process of obtaining the emotion information based on the feature image includes: The feature image is input into the Transformer encoder to obtain the feature vector corresponding to the target image. The Transformer encoder includes a multi-head self-attention module, a multilayer perceptron, and a layer normalization module. The feature vector is input into a fully connected layer to obtain the emotional information represented by facial information in the target image; The training of the emotion classification network includes: Obtain a training set, which includes multiple training images, each of which includes facial information and pre-labeled emotion tags corresponding to the facial information. For any target training image in the training set, the target training image is input into the RedNet feature extractor in the initial emotion classification network to obtain the feature image of the target training image; The feature image of the target training image is input into the Transformer encoder to obtain the feature vector corresponding to the target training image; The feature vector corresponding to the target training image is input into a fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image. Based on the predicted labels and the pre-labeled emotion labels of the target training images, the parameters in the emotion classification network are adjusted to obtain the trained emotion classification network. The fully connected layer includes an attention factor. The step of inputting the feature vector corresponding to the target training image into the fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image includes: The feature vector corresponding to the target training image is input into a fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image, as well as the weight information of the target training image. The step of adjusting the parameters in the emotion classification network based on the predicted labels and the pre-labeled emotion labels of the training images includes: Based on the predicted labels, the pre-labeled emotion labels on the target training images, and the weight information of the target training images, the parameters in the emotion classification network are adjusted using the cross-entropy loss function and regularization loss.
2. The method according to claim 1, characterized in that, The method further includes: Obtain a test set, which includes multiple test images, each of which includes facial information and an emotion label pre-annotated to the facial information; For any target test image in the test set, the target test image is input into the RedNet feature extractor in the trained emotion classification network to obtain the feature image of the target test image; The feature image of the target test image is input into the Transformer encoder to obtain the feature vector corresponding to the target test image; The feature vector corresponding to the target test image is input into the MC-dropout layer to determine the uncertainty information of the target test image; Determine whether the uncertainty information of the multiple test images meets a preset rule. If the preset rule is met, the trained emotion classification network is used as the trained emotion classification network.
3. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire a target image including facial information; The emotion determination module is used to input the target image into a pre-trained emotion classification network to obtain emotion information represented by facial information in the target image; The emotion classification network includes a RedNet feature extractor composed of involution operators. The RedNet feature extractor is used to obtain a feature image based on the target image, and to obtain the emotion information based on the feature image. The emotion determination module is specifically used for: The feature image is input into the Transformer encoder to obtain the feature vector corresponding to the target image. The Transformer encoder includes a multi-head self-attention module, a multilayer perceptron, and a layer normalization module. The feature vector is input into a fully connected layer to obtain the emotional information represented by facial information in the target image; The device includes: The second acquisition module is used to acquire a training set, which includes multiple training images. Each training image includes facial information and an emotion label pre-annotated to the facial information. The feature extraction module, for any target training image in the training set, inputs the target training image into the RedNet feature extractor in the initial emotion classification network to obtain the feature image of the target training image; The feature vector determination module is used to input the feature image of the target training image into the Transformer encoder to obtain the feature vector corresponding to the target training image; The prediction module is used to input the feature vector corresponding to the target training image into the fully connected layer to obtain the predicted label corresponding to the emotion information represented by the facial information in the target training image; An adjustment module is used to adjust the parameters in the emotion classification network based on the predicted labels and the emotion labels pre-labeled on the target training image, so as to obtain the trained emotion classification network. The fully connected layer includes an attention factor; The prediction module is specifically used to input the feature vector corresponding to the target training image into a fully connected layer to obtain the prediction label corresponding to the emotion information represented by the facial information in the target training image, as well as the weight information of the target training image. The adjustment module is specifically used to adjust the parameters in the emotion classification network based on the predicted label, the pre-labeled emotion label of the target training image, and the weight information of the target training image, using the cross-entropy loss function and regularization loss.
4. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in claim 1 or 2.
5. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method of claim 1 or 2.