Expression recognition model and training method, recognition method, device and computing device
By using multiple convolutional layers of different depths for supervision and training in the expression recognition neural network model, the problem that shallow convolutional layer features in the existing technology is difficult to participate in training, and the accuracy and resilience of facial expression recognition are improved. Great.
Patent Information
- Application Number
- CN202011217953.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-04
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-11-04
AI Technical Summary
The existing expression recognition neural network model is only supervised and trained based on the last convolutional layer, which makes it difficult for the texture features of the shallow convolutional layer to participate in training, resulting in low recognition rate.
Multiple convolutional layers of different depths are used to determine multiple classification loss functions for supervision and training of neural network models, so that the model can be supervised separately at convolutional layers of different depths, and the information extracted from different depths is used to adjust parameters to improve identification accuracy. Rate and robustness.
It significantly improves the accuracy and robustness of facial expression recognition in complex environments, and makes full use of feature information of different depth layers of neural network models.
Smart Images

Figure CN112183491B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, specifically the field of computer vision, deep learning and image processing technology, and in particular to expression recognition models and training methods, recognition methods, devices, computing equipment and media. Background Art
[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include fields such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions. Artificial intelligence is being used more and more widely in various fields, such as face recognition.
[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention
[0004] According to one aspect of the present disclosure, a training method for an expression recognition neural network model is provided, wherein the expression recognition neural network model includes a main neural network model and at least one branch neural network model, wherein the main neural network model includes a plurality of first convolutional layers connected in sequence, wherein the training method includes: obtaining a sample image containing a human face, and marking the real expression category of the human face contained in the sample image and its real category confidence; determining at least one first convolutional layer from a plurality of first convolutional layers connected in sequence except the last first convolutional layer close to the output side; in response to the sample image being input into the main neural network model, the main neural network model outputs the main predicted expression category of the human face contained in the sample image and its main category confidence; Predicted category confidence; for each branch neural network model, in response to a feature map of one of the corresponding outputs in at least one first convolutional layer input into the branch neural network model, the branch neural network model outputs a branch predicted expression category of the face contained in the sample image and its branch predicted category confidence; based on the main predicted expression category and its main predicted category confidence, as well as the real expression category and its real category confidence, determine the main loss value; for each branch neural network model, based on the branch predicted expression category and its branch predicted category confidence, as well as the real expression category and its real category confidence, determine the branch loss value; and adjust the parameters of the neural network model based on at least the main loss value and at least one branch loss value.
[0005] According to another aspect of the present disclosure, there is also provided an expression recognition method, comprising: inputting an image to be recognized containing a human face into an expression recognition neural network model, the expression recognition neural network model comprising a main neural network model and at least one branch neural network model, the main neural network model comprising a plurality of first convolutional layers connected in sequence; in response to the image to be recognized being input into the main neural network model, a main feature vector output by the main neural network model is obtained; in response to a feature map outputted by a corresponding one of at least one first convolutional layer being input into a branch neural network model, a branch feature vector outputted by the branch neural network model is obtained, wherein at least one first convolutional layer is determined from a plurality of first convolutional layers connected in sequence except the last first convolutional layer close to the output side; and determining the expression category of the human face contained in the image to be recognized and its category confidence based on the main feature vector and the branch feature vector.
[0006] According to another aspect of the present disclosure, there is also provided an expression recognition neural network model, comprising: a main neural network model, wherein the main neural network model comprises a plurality of first convolutional layers connected in sequence, and the main neural network model is configured to output a main feature vector in response to an image to be recognized being input into the main neural network model; and at least one branch neural network model, each branch neural network model being configured to output a branch feature vector in response to a feature map outputted from a corresponding one of the at least one first convolutional layers being input into the branch neural network model, wherein at least one first convolutional layer is determined from a plurality of first convolutional layers connected in sequence except the last first convolutional layer close to the output side.
[0007] According to another aspect of the present disclosure, there is also provided an expression recognition device, comprising: an expression recognition neural network model, comprising a main neural network model and at least one branch neural network model, the main neural network model comprising a plurality of first convolutional layers connected in sequence; an input unit, configured to input an image to be recognized containing a human face into the expression recognition neural network model; a first acquisition unit, configured to obtain a main feature vector output by the main neural network model in response to the image to be recognized being input into the main neural network model; a second acquisition unit, configured to input a feature map output by a corresponding one of at least one first convolutional layer into the branch neural network model, and obtain a branch feature vector output by the branch neural network model, wherein at least one first convolutional layer is determined from a plurality of first convolutional layers connected in sequence except the last first convolutional layer close to the output side; and a determination unit, configured to determine the expression category of the human face contained in the image to be recognized and its category confidence based on the main feature vector and the branch feature vector.
[0008] According to another aspect of the present disclosure, a computing device is also provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above-mentioned training method and / or the above-mentioned recognition method.
[0009] According to another aspect of the present disclosure, a computer-readable storage medium storing a program is also provided. The program includes instructions. When the instructions are executed by a processor of a computing device, the computing device executes the above-mentioned training method and / or the above-mentioned recognition method.
[0010] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned training method and / or the steps of the above-mentioned recognition method are implemented.
[0011] The present invention performs supervised training of a neural network model based on multiple classification loss functions determined by multiple convolutional layers of different depths in the neural network model. It can perform supervision separately at convolutional layers of different depths in the neural network model, which can not only make it easier for the neural network model to learn expression information, but also easier to converge. At the same time, it makes full use of the different information extracted by convolutional layers of different depths of the neural network model, which can greatly improve the accuracy and robustness of facial expression recognition in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0013] Figure 1 is a flow chart illustrating a method for training an expression recognition neural network model according to an exemplary embodiment;
[0014] Figure 2 is a flow chart illustrating an expression recognition method according to an exemplary embodiment of the present invention;
[0015] Figure 3 is a schematic block diagram showing a composition of an expression recognition neural network model according to an exemplary embodiment;
[0016] Figure 4 is a schematic block diagram showing a composition of an expression recognition apparatus according to an exemplary embodiment;
[0017] Figure 5 is a block diagram illustrating the structure of an exemplary computing device that can be used in accordance with the exemplary embodiments. DETAILED DESCRIPTION
[0018] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.
[0019] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.
[0020] As one aspect of the application of artificial intelligence technology, computers can be used to simulate human thinking processes based on trained neural network models to identify the information contained in input images. The neural network model can include a number of sequentially connected convolutional layers, wherein the convolutional layers closer to the input side of the neural network model are shallower, and the feature maps they output contain visual features of the input image, such as texture, edge, and chromaticity, and the convolutional layers farther from the input side of the neural network model are deeper, and the feature maps they output contain semantic features of the input image.
[0021] In one aspect of image recognition based on neural networks, facial expressions in input images containing human faces can be recognized.
[0022] Facial expressions can be expressed as deformation or distortion of facial muscles. For example, a happy expression is expressed as the cheek muscles pulling upward. In an image containing a face, this is reflected as a change in facial texture information caused by the deformation or distortion of muscles. In a neural network model, texture features are usually reflected in the feature map output by the shallow convolution layer.
[0023] In the related art, during the training of the expression recognition neural network model, supervised training is performed only based on the feature map output by the last convolution layer of the neural network model, while the texture features contained in the feature map output by the shallow convolution layer cannot directly participate in the supervised training, resulting in a serious loss of texture features important for expression classification, and the parameters of the shallow convolution layer are difficult to be effectively supervised, resulting in a low recognition rate of the trained expression recognition network model. Based on this, the present disclosure proposes an expression recognition model and a training method, a recognition method, an apparatus and a computing device, which perform supervised training of the neural network model based on multiple classification loss functions determined by multiple convolution layers of different depths in the neural network model, and can perform supervision separately at convolution layers of different depths of the neural network model, which can not only make the neural network model easier to learn expression information, but also easier to converge, and at the same time, make full use of the different information extracted by the convolution layers of different depths of the neural network model, which can greatly improve the accuracy and robustness of facial expression recognition in complex environments.
[0024] The training method of the expression recognition neural network model disclosed in the present invention will be further described below in conjunction with the accompanying drawings.
[0025] Figure 1 1 is a flow chart showing a method for training an expression recognition neural network model according to an exemplary embodiment of the present disclosure, wherein the neural network model includes a main neural network model and at least one branch neural network model, wherein the main neural network model includes a plurality of first convolutional layers connected in sequence. Figure 1As shown, the training method includes: step S101, obtaining a sample image containing a human face, and marking the real expression category of the human face contained in the sample image and its real category confidence; step S102, determining at least one first convolutional layer from a plurality of first convolutional layers other than the last first convolutional layer close to the output side among the plurality of first convolutional layers connected in sequence; step S103, in response to the sample image inputting the main neural network model, the main neural network model outputs the main predicted expression category of the human face contained in the sample image and its main predicted category confidence; step S104, for each branch neural network model, in response to the corresponding expression category in at least one first convolutional layer One of the output feature maps is input into the branch neural network model, and the branch neural network model outputs the branch predicted expression category and its branch predicted category confidence of the face contained in the sample image; step S105, based on the main predicted expression category and its main predicted category confidence, and the real expression category and its real category confidence, determine the main loss value; step S106, for each branch neural network model, based on the branch predicted expression category and its branch predicted category confidence, and the real expression category and its real category confidence, determine the branch loss value; and step S107, adjust the parameters of the neural network model based on at least the main loss value and at least one branch loss value. Thus, in the process of training the neural network, the neural network model can learn the features output by multiple convolutional layers of different depths, and can supervise the convolutional layers of different depths of the neural network model respectively, which can not only make the neural network model easier to learn expression information, but also easier to converge, and at the same time make full use of the different information extracted by the convolutional layers of different depths of the neural network model, which can greatly improve the accuracy and robustness of facial expression recognition in complex environments.
[0026] Among them, the neural network model can be a VGG11 convolutional neural network model, or other types of VGG convolutional neural networks, or other neural network models such as LeNet, AlexNet, NiN, GooLeNet, ResNet, DenseNet, etc., which are not limited here.
[0027] With respect to step S101, the sample image containing a human face may include one human face or multiple human faces. When the sample image includes only one human face, a true expression category and a corresponding true category confidence may be annotated for the one human face included in the sample image. When the sample image includes multiple human faces, a corresponding true expression category and a true category confidence may be annotated for each human face included in the sample image.
[0028] According to some embodiments, real expression categories may include anger, disgust, fear, happiness, sadness, surprise, and neutral.
[0029] With respect to step S102, according to some embodiments, at least one first convolutional layer may include two or more first convolutional layers, and at least one of the at least one first convolutional layer is close to the input side of the main neural network model, that is, a shallow convolutional layer. Thus, during the training of the neural network, the neural network model can learn the texture features of the sample image based on the first convolutional layer close to the input side of the main neural network model, thereby improving the recognition accuracy of the neural network for expressions.
[0030] In one embodiment, when the main neural network model is a VGG11 convolutional neural network model, at least one first convolutional layer may include the third convolutional layer and the eighth convolutional layer of the VGG11 convolutional neural network model. Thus, the "at least one first convolutional layer" in step S102 may include the third convolutional layer and the eighth convolutional layer of the VGG11 convolutional neural network model. Thus, the output features of shallow, middle and high convolutional layers can be selected from multiple convolutional layers in the neural network model for supervised learning. It should be noted that here, the VGG11 convolutional neural network model is used as an example to illustrate how to determine "at least one first convolutional layer", and the main neural network model disclosed in the present invention is not limited to the VGG11 convolutional neural network model.
[0031] For step S107, according to some embodiments, a first weighted sum of the main loss value and at least one branch loss value can be calculated, and the parameters of the neural network model can be adjusted based on at least the first weighted sum. Thus, in different application scenarios, based on the degree of influence of the features output by the convolutional layers of different depths in the neural network model on the recognition accuracy of the neural network model, different weights can be assigned to the loss values based on the convolutional layers of different depths for supervised training, thereby improving the recognition accuracy of the neural network. In the application of expression recognition of images containing human faces, preferably, a larger weight can be assigned to at least one branch loss value based on the shallow convolutional layer to enhance the learning of the neural network model on the texture features of the image and improve the accuracy of the neural network model in recognizing the age of the human face.
[0032] According to some embodiments, the branch neural network model may include a plurality of second convolutional layers connected in sequence, and the training method of the expression recognition neural network model may also include determining a central loss value based on a plurality of feature maps output by the last first convolutional layer closest to the output side of the main neural network model, a plurality of feature maps output by the last second convolutional layer closest to the output side of each of at least one branch neural network model, and the category feature centers corresponding to each of the plurality of feature maps, wherein adjusting the parameters of the neural network model based at least on the main loss value and at least one branch loss value includes: calculating a second weighted sum of the main loss value, at least one branch loss value, and the central loss value; and adjusting the parameters of the neural network model based on the second weighted sum. Thus, by auxiliary supervision of the training of the face recognition neural network through the central loss value, the difference in the feature maps output by the sample images of the same expression category can be reduced, thereby further improving the accuracy and robustness of the face recognition neural network in recognizing the age of the face.
[0033] In an exemplary embodiment, the center loss value may be calculated as follows:
[0034]
[0035] In formula (1), represents the center loss value, m represents the number of sample images of the current batch of inputs, i represents the number of sample images in the current batch, 1≤i≤m, i is a positive integer, x i The total feature map representing the i-th sample image in the current batch is composed of multiple feature maps output by the last first convolutional layer closest to the output side of the main neural network model and multiple feature maps output by the last second convolutional layer closest to the output side of each of at least one branch neural network model, c yi Represents the true expression category y corresponding to the i-th sample image in the current batch i The category feature center of represents the operation of the square of the bi-norm. The true expression categories of different sample images can be the same or different.
[0036] Among them, the category feature center c yi With the total feature map x i has the same number of channels as x and has the same size as the feature map on the corresponding channel. i -c yi The calculation is based on the total feature map x i and category feature center c yi It is performed on the feature map of each corresponding channel.
[0037] For the total feature map x i According to some embodiments, the total feature map x iIt can be obtained by superimposing multiple feature maps output by the last first convolutional layer closest to the output side of the main neural network model and multiple feature maps output by the last second convolutional layer closest to the output side of each branch neural network model in at least one branch neural network model. In other words, the number of feature maps of the total feature map can be the sum of the number of feature maps output by the last first convolutional layer closest to the output side of the main neural network model and the number of feature maps output by the last second convolutional layer closest to the output side of each branch neural network model in at least one branch neural network model.
[0038] For the category feature center c yi , different real expression categories y i Corresponding to different category feature centers c yi According to some embodiments, the category feature center c yi It can be but is not limited to being determined by the average value of multiple feature maps obtained from the output of a neural network model based on multiple sample images of the same real expression category in a previous batch.
[0039] According to some embodiments, the sample images containing human faces may be preprocessed before the sample images are input into the main neural network model, thereby optimizing the input sample images and improving the training effect of the neural network model.
[0040] According to some embodiments, preprocessing a sample image containing a human face may include: determining the area where the human face in the sample image is located; determining the coordinate values of multiple key points of the human face based on the area where the human face is located; generating an affine matrix based on the coordinate values of the multiple key points; performing an affine transformation on the sample image based on the affine matrix to align the human face in the sample image, wherein the affine matrix includes the angle parameter between the line connecting the key points of the corners of the eyes and the horizontal direction. Thus, by aligning the human face contained in the input sample image, the training effect of the neural network model can be improved.
[0041] The calculation method of the affine transformation matrix and the affine transformation method of the image are common knowledge in the art and are not limited here.
[0042] In one implementation, with respect to “determining the region where the human face in the sample image is located”, the human face in the sample image may be detected using a face detection model to determine the region where the human face in the sample image is located.
[0043] In one embodiment, for “determining the coordinate values of multiple key points of the face”, the coordinate values of the key points in the face region can be determined by a key point detection model. Preferably, the multiple key points can be 72 key points included in the contour and facial features of the face.
[0044] In one embodiment, in the process of performing affine transformation on the sample image based on the affine matrix to align the face in the sample image, the sample image can be cropped to obtain a sample image containing only the face area, and the cropped sample image can be adjusted to the same size of 128x128.
[0045] According to some embodiments, preprocessing the sample image containing the face may further include: normalizing the sample image after performing affine transformation on the sample image, thereby reducing the difference between the pixel values of the sample image and improving the training effect of the neural network model.
[0046] In one implementation, the normalization method may be: subtract 128 from the pixel value of each pixel of the sample image, and then divide the value by 256, so that the pixel value of each pixel is between [-0.5, 0.5].
[0047] According to some embodiments, preprocessing the sample images containing human faces may further include: performing random data enhancement processing on the sample images after normalizing the sample images, thereby increasing the number of sample images for training and improving the anti-interference ability of the neural network model.
[0048] In one embodiment, the random data enhancement processing performed on the sample image may include one or more of random flipping processing, random translation processing, random scaling processing, random grayscale processing, random gamma transformation processing, and random addition of Gaussian white noise.
[0049] The training method of the neural network model for expression recognition proposed in the present disclosure is only an exemplary application of the training method of the neural network model proposed, but is not limited thereto. The application fields of the neural network model training method proposed in the present disclosure may also include other application fields that require supervised training based on the output features of the shallow convolutional layer of the neural network.
[0050] The following will be combined with the attached Figure 2 The expression recognition method disclosed in the present invention is further described.
[0051] Figure 2 is a flow chart showing a method for facial expression recognition according to an exemplary embodiment of the present disclosure, Figure 2As shown, the expression recognition method may include: step S201, inputting an image to be recognized containing a human face into an expression recognition neural network model, the expression recognition neural network model includes a main neural network model and at least one branch neural network model, and the main neural network model includes multiple first convolutional layers connected in sequence; step S202, in response to the image to be recognized being input into the main neural network model, obtaining the main feature vector output by the main neural network model; step S203, in response to the feature map of one of the corresponding outputs of at least one first convolutional layer being input into the branch neural network model, obtaining the branch feature vector output by the branch neural network model, wherein at least one first convolutional layer is determined from multiple first convolutional layers connected in sequence except the last first convolutional layer close to the output side; step S204, based on the main feature vector and the branch feature vector, determining the expression category and category confidence of the human face contained in the image to be recognized. Thus, expression recognition can be performed based on the output features of the convolutional layers of different depths of the neural network model, so that the neural network model can more easily obtain information for expression recognition, and make full use of the different information extracted by the convolutional layers of different depths of the neural network model, which can improve the accuracy and robustness of facial expression recognition in complex environments.
[0052] According to some embodiments, based on the main feature vector and the branch feature vector, determining the expression category and its category confidence of the face contained in the image to be identified includes: channel superposition of the main feature vector and the branch feature vector; and based on the feature vector obtained by channel superposition, determining the expression category and its category confidence of the face contained in the image to be identified. Thus, a feature vector containing the output features of multiple depths of convolutional layers in the neural network model can be obtained, and expression recognition is performed based on the feature vector.
[0053] For example, at least one branch neural network model includes a first branch neural network model and a second branch neural network model, the first branch neural network model outputs a 1×1×7 first branch feature vector, the second branch neural network model outputs a 1×1×7 second branch feature vector, and the main neural network model outputs a 1×1×7 main feature vector. In the above case, the first branch feature vector, the second branch feature vector and the main feature vector are channel-superimposed to form a feature vector of size 1×1×21, which may include the features of the first branch feature vector, the second branch feature vector and the main feature vector. Based on the expression category and category confidence of the face contained in the image to be identified determined by the 1×1×21 feature vector, the output features of the convolutional layers of different depths of the neural network model are integrated, and the facial features extracted from different layers of the model are fully utilized, which can greatly improve the accuracy and robustness of facial expression recognition in complex environments.
[0054] According to some embodiments, before the image to be recognized containing a human face is input into the facial expression recognition neural network model, the image to be recognized is preprocessed, thereby optimizing the input image to be recognized and improving the recognition effect of facial expressions.
[0055] According to some embodiments, preprocessing an image to be recognized that includes a face includes: determining the area where the face in the image to be recognized is located; determining the coordinate values of multiple key points of the face based on the area where the face is located; generating an affine matrix based on the coordinate values of the multiple key points; performing an affine transformation on the image to be recognized based on the affine matrix to align the face in the image to be recognized, wherein the affine matrix includes the angle parameter between the line connecting the key points of the corners of the eyes and the horizontal direction. Thus, the face contained in the input image to be recognized is aligned to improve the recognition effect of facial expressions.
[0056] In one implementation, with respect to “determining the region where the face in the image to be recognized is located”, the face in the image to be recognized may be detected using a face detection model to determine the region where the face in the image to be recognized is located.
[0057] In one embodiment, for “determining the coordinate values of multiple key points of the face”, the coordinate values of the key points in the face region can be determined by a key point detection model. Preferably, the multiple key points can be 72 key points included in the contour and facial features of the face.
[0058] In one embodiment, in the process of performing an affine transformation on the image to be identified based on an affine matrix to align the face in the image to be identified, the image to be identified can be cropped to obtain an image to be identified that only contains the face area, and the cropped image to be identified can be adjusted to the same size of 128x128.
[0059] According to some embodiments, preprocessing the image to be recognized containing a human face may further include: normalizing the image to be recognized after performing an affine transformation on the image to be recognized, thereby reducing the difference between pixel values of the image to be recognized and improving the recognition effect of facial expressions.
[0060] In one implementation, the normalization method may be: subtract 128 from the pixel value of each pixel of the image to be recognized, and then divide the value by 256, so that the pixel value of each pixel is between [-0.5, 0.5].
[0061] The expression recognition method disclosed in the present invention can be applied in, but is not limited to, the fields of visual interaction, intelligent control, assisted driving, distance education, and precise advertising delivery.
[0062] According to another aspect of the present disclosure, an expression recognition neural network model 300 is also provided. Figure 3An exemplary expression recognition neural network model 300 is shown. Figure 3 As shown, the expression recognition neural network model 300 includes: a main neural network model 301, wherein the main neural network model includes a plurality of first convolutional layers connected in sequence, and the main neural network model is configured to respond to an image to be recognized input into the main neural network model and output a main feature vector; and at least one branch neural network model 302, each branch neural network model is configured to respond to a feature map outputted from a corresponding one of at least one first convolutional layer input into the branch neural network model and output a branch feature vector, wherein at least one first convolutional layer is determined from a plurality of first convolutional layers connected in sequence except the last first convolutional layer close to the output side.
[0063] According to some embodiments, the main neural network model may be, but is not limited to, a VGG11 convolutional neural network model, and at least one first convolutional layer includes the third convolutional layer and the eighth convolutional layer of the VGG11 convolutional neural network model. The above-mentioned neural network model applied to expression recognition proposed in the present disclosure is only an exemplary application field of the proposed neural network model, but is not limited thereto.
[0064] The application fields of the neural network model proposed in the present disclosure may also include other application fields that require feature recognition based on the output features of the shallow convolutional layer of the neural network.
[0065] According to another aspect of the present disclosure, Figure 4 As shown, an expression recognition device 400 is also provided, which may include: an expression recognition neural network model 401, including a main neural network model and at least one branch neural network model, the main neural network model including a plurality of first convolutional layers connected in sequence; an input unit 402, configured to input an image to be recognized containing a human face into the expression recognition neural network model; a first acquisition unit 403, configured to obtain a main feature vector output by the main neural network model in response to the image to be recognized being input into the main neural network model; a second acquisition unit 404, configured to input a branch neural network model in response to a feature map outputted by a corresponding one of at least one first convolutional layer, to obtain a branch feature vector outputted by the branch neural network model, wherein at least one first convolutional layer is determined from a plurality of first convolutional layers connected in sequence except the last first convolutional layer close to the output side; and a determination unit 405, configured to determine the expression category of the human face contained in the image to be recognized and its category confidence based on the main feature vector and the branch feature vector.
[0066] According to some embodiments, the determination unit may include: a superposition subunit configured to perform channel superposition on the main feature vector and the branch feature vector; and a determination subunit configured to determine the expression category and category confidence of the face contained in the image to be identified based on the feature vector obtained by channel superposition.
[0067] According to some embodiments, the recognition device may further include: a preprocessing unit configured to preprocess the image to be recognized containing a human face before inputting the image to be recognized containing a human face into the facial expression recognition neural network model.
[0068] According to another aspect of the present disclosure, a computing device is also provided, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above-mentioned training method and / or the above-mentioned recognition method.
[0069] According to another aspect of the present disclosure, a computer-readable storage medium storing a program is also provided. The program includes instructions. When the instructions are executed by a processor of a computing device, the computing device executes the above-mentioned training method and / or the above-mentioned recognition method.
[0070] See also Figure 5 As shown, a computing device 5000 will now be described, which is an example of a hardware device (electronic device) that can be applied to various aspects of the present disclosure. The computing device 5000 can be any machine configured to perform processing and / or calculations, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a robot, a smart phone, a vehicle-mounted computer, or any combination thereof. The above-mentioned generation method can be implemented in whole or in part by the computing device 5000 or a similar device or system.
[0071] The software elements (programs) may be located in the working memory 5014, including but not limited to an operating system 5016, one or more application programs 5018, a driver and / or other data and code. Instructions for executing the above methods and steps may be included in one or more application programs 5018, and one or more of the training methods and the recognition methods may be implemented by the processor 5004 reading and executing the instructions of one or more application programs 5018. More specifically, in the above training method, steps S101 to S107 may be implemented, for example, by the processor 5004 executing the application program 5018 having the instructions of steps S101 to S107. In the above recognition method, steps S201 to S204 may be implemented, for example, by the processor 5004 executing the application program 5018 having the instructions of steps S201 to S204. In addition, other steps of one or more of the above training methods and the recognition method may be implemented, for example, by the processor 5004 executing the application program 5018 having the instructions for executing the corresponding steps. The executable code or source code of the instructions of the software element (program) can be stored in a non-transitory computer-readable storage medium (such as the storage device 5010 described above), and when executed, can be stored in the working memory 5014 (possibly compiled and / or installed). The executable code or source code of the instructions of the software element (program) can also be downloaded from a remote location.
[0072] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages (such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.
[0073] It should also be understood that the aforementioned method can be implemented through a server-client model. For example, the client can receive data input by the user and send the data to the server. The client can also receive data input by the user, perform part of the processing in the aforementioned method, and send the processed data to the server. The server can receive data from the client, and execute the aforementioned method or another part of the aforementioned method, and return the execution result to the client. The client can receive the execution result of the method from the server and present it to the user, for example, through an output device. The server can be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology.
[0074] It should also be understood that the components of computing device 5000 can be distributed across a network. For example, some processing can be performed using one processor while other processing can be performed by another processor remote from the one processor. Other components of computing system 5000 can also be similarly distributed. In this way, computing device 5000 can be interpreted as a distributed computing system that performs processing at multiple locations.
[0075] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.
Claims
1. A training method for an expression recognition neural network model, wherein the expression recognition neural network model comprises a main neural network model and at least one branch neural network model, wherein the main neural network model comprises a plurality of first convolutional layers connected in sequence, wherein: The training method comprises: Acquire a sample image containing a human face, and annotate the real expression category of the human face contained in the sample image and its real category confidence; Determine at least one first convolutional layer from among the plurality of sequentially connected first convolutional layers except for the last first convolutional layer close to the output side; In response to the sample image being input into the main neural network model, the main neural network model outputs a main predicted expression category of a human face contained in the sample image and its main predicted category confidence; For each of the branch neural network models, in response to a feature map outputted from a corresponding one of the at least one first convolutional layer, the branch neural network model outputs a branch predicted expression category of a face contained in the sample image and a confidence level of the branch predicted category; Determining a main loss value based on the main predicted expression category and its main predicted category confidence, and the real expression category and its real category confidence; For each of the branch neural network models, determining a branch loss value based on the branch predicted expression category and its branch predicted category confidence, and the true expression category and its true category confidence; and Parameters of the neural network model are adjusted based at least on the main loss value and at least one of the branch loss values.
2. The training method according to claim 1, wherein: The adjusting the parameters of the neural network model based at least on the main loss value and at least one of the branch loss values comprises: calculating a first weighted sum of the main loss value and the at least one branch loss value; and Parameters of the neural network model are adjusted based at least on the first weighted sum.
3. The training method according to claim 1, wherein: The at least one first convolutional layer includes two or more first convolutional layers, and at least one of the at least one first convolutional layers is close to the input side of the main neural network model.
4. The training method according to claim 3, wherein: The main neural network model is a VGG11 convolutional neural network model, and the at least one first convolutional layer includes the third convolutional layer and the eighth convolutional layer of the VGG11 convolutional neural network model.
5. The training method according to claim 1, wherein: The branched neural network model includes a plurality of second convolutional layers connected in sequence; The training method further comprises: Determine a central loss value based on a plurality of feature maps output by the last first convolutional layer closest to the output side of the main neural network model, a plurality of feature maps output by the last second convolutional layer closest to the output side of each of the at least one branch neural network model, and the corresponding category feature centers of the plurality of feature maps, Wherein, adjusting the parameters of the neural network model based at least on the main loss value and at least one of the branch loss values comprises: Calculating a second weighted sum of the main loss value, the at least one branch loss value, and the center loss value; Based on the second weighted sum, parameters of the neural network model are adjusted.
6. The training method according to any one of claims 1 to 5, further comprising: Before the sample image is input into the main neural network model, the sample image containing the face is preprocessed.
7. The training method according to claim 6, wherein: The preprocessing of the sample image containing the face includes: Determine the area where the human face in the sample image is located; Determining coordinate values of a plurality of key points of the face based on the area where the face is located; Based on the coordinate values of the plurality of key points, generating an affine matrix; and Affine transformation is performed on the sample image based on the affine matrix to align faces in the sample image, wherein the affine matrix includes angle parameters between a line connecting key points of eye corners and a horizontal direction.
8. The training method according to claim 7, wherein: The preprocessing of the sample image containing the face also includes: After performing affine transformation on the sample image, normalization processing is performed on the sample image.
9. The training method according to claim 8, wherein: The preprocessing of the sample image containing the face also includes: After the sample image is normalized, random data enhancement is performed on the sample image.
10. A method for facial expression recognition, comprising: Inputting an image to be recognized containing a human face into an expression recognition neural network model, wherein the expression recognition neural network model includes a main neural network model and at least one branch neural network model, wherein the main neural network model includes a plurality of first convolutional layers connected in sequence; In response to the image to be recognized being input into the main neural network model, obtaining a main feature vector output by the main neural network model; In response to a feature map of a corresponding one of the outputs of at least one of the first convolutional layers being input into the branch neural network model, a branch feature vector output by the branch neural network model is obtained, wherein the at least one first convolutional layer is determined from a plurality of first convolutional layers other than the last first convolutional layer close to the output side among the plurality of sequentially connected first convolutional layers; and Based on the main feature vector and the branch feature vector, the expression category and category confidence of the face contained in the image to be recognized are determined.
11. The facial expression recognition method according to claim 10, wherein: The step of determining the expression category and category confidence of the face contained in the image to be recognized based on the main feature vector and the branch feature vector comprises: Perform channel superposition on the main feature vector and the branch feature vector; and Based on the feature vector obtained by channel superposition, the expression category and category confidence of the face contained in the image to be recognized are determined.
12. The facial expression recognition method according to claim 10 or 11, further comprising: Before inputting the image to be recognized containing a human face into the expression recognition neural network model, the image to be recognized is preprocessed.
13. An expression recognition device, comprising: An expression recognition neural network model, comprising a main neural network model and at least one branch neural network model, wherein the main neural network model comprises a plurality of first convolutional layers connected in sequence; An input unit, configured to input an image to be recognized containing a human face into the expression recognition neural network model; A first acquisition unit is configured to acquire a main feature vector output by the main neural network model in response to the image to be recognized being input into the main neural network model; A second acquisition unit is configured to input a feature map of one of the outputs of at least one of the first convolutional layers into the branch neural network model, and acquire a branch feature vector output by the branch neural network model, wherein the at least one first convolutional layer is determined from a plurality of first convolutional layers other than the last first convolutional layer close to the output side among the plurality of sequentially connected first convolutional layers; as well as A determination unit is configured to determine the expression category and category confidence of the face contained in the image to be recognized based on the main feature vector and the branch feature vector.
14. The facial expression recognition device according to claim 13, wherein: The determining unit comprises: a superposition subunit, configured to perform channel superposition on the main feature vector and the branch feature vector; and The determination subunit is configured to determine the expression category and category confidence of the face contained in the image to be identified based on the feature vector obtained by channel superposition.
15. The facial expression recognition device according to claim 13 or 14, further comprising: The preprocessing unit is configured to preprocess the image to be recognized containing a human face before inputting the image to be recognized containing a human face into the expression recognition neural network model.
16. A computing device comprising: processor; as well as A memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor is caused to perform the training method according to any one of claims 1-9 and / or the recognition method according to any one of claims 10-12.
17. A computer-readable storage medium storing a program, wherein the program comprises instructions, which, when executed by a processor of a computing device, cause the computing device to perform the training method according to any one of claims 1 to 9 and / or the recognition method according to any one of claims 10 to 12.
