Emotion recognition model training method and device, storage medium, and terminal

Abstract

A training method, apparatus, storage medium, and terminal for an emotion recognition model are disclosed. The method includes: acquiring sample facial feature vectors, sample body feature vectors, and sample image feature vectors based on sample images; calculating a first weight corresponding to the sample facial feature vector and a second weight corresponding to the sample body feature vector based on a first sample feature vector and a second sample feature vector, wherein the first sample feature vector is obtained by fusing the sample facial feature vector and the sample image feature vector, and the second sample feature vector is obtained by fusing the sample body feature vector and the sample image feature vector; fusing the sample facial feature vector and the sample body feature vector according to the first weight and the second weight to obtain a third sample feature vector; calculating a prediction result of the emotion category of a first person based on the third sample feature vector; calculating a loss based on the prediction result and the emotion category label of the first person, and updating the feature fusion module according to the loss.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method and apparatus, storage medium and terminal for an emotion recognition model. Background Technology

[0002] Image recognition and classification technology is one of the most widely used artificial intelligence technologies today. Image recognition technology refers to the technique of identifying objects in images to recognize targets and objects of various patterns. Emotion recognition is one of the tasks of image recognition and classification.

[0003] Emotion recognition is an important area for computers to understand human emotions, and it can determine the emotions of faces in a specific scene. Traditional methods identify the emotions associated with facial images by recognizing key points within those images. However, the accuracy of current emotion recognition still needs further improvement. Summary of the Invention

[0004] The technical objective of this application is to provide a training method, apparatus, storage medium, and terminal for an emotion recognition model, so as to improve the accuracy of emotion recognition.

[0005] In view of this, embodiments of this application provide a training method for an emotion recognition model. The emotion recognition model includes a feature fusion module, which includes a weight calculation unit and a fusion unit. The method includes: obtaining a sample facial feature vector, a sample body feature vector, and a sample image feature vector based on a sample image, wherein the sample facial feature vector is obtained by extracting features from the facial region of a first person in the sample image, the sample body feature vector is obtained by extracting features from the body region of the first person in the sample image, and the sample image feature vector is obtained by extracting features from the sample image; and using the weight calculation unit, calculating the weight of the sample image based on the first sample feature vector and the second sample feature vector. The model uses a first weight corresponding to the facial feature vector and a second weight corresponding to the sample body feature vector. The first sample feature vector is obtained by fusing the sample facial feature vector and the sample image feature vector, and the second sample feature vector is obtained by fusing the sample body feature vector and the sample image feature vector. A fusion unit is used to fuse the sample facial feature vector and the sample body feature vector according to the first weight and the second weight to obtain a third sample feature vector. Based on the third sample feature vector, the prediction result of the emotion category of the first person is calculated. The loss is calculated based on the prediction result and the emotion category label of the first person, and the feature fusion module is updated according to the loss until the emotion recognition model converges.

[0006] Optionally, before obtaining the sample facial feature vector, sample body feature vector, and sample image feature vector based on the sample image, the method further includes: calculating the ratio of the size of the facial region to the size of the body region of the first person; determining whether the ratio is less than a preset threshold; if the determination result is yes, then obtaining the sample facial feature vector, sample body feature vector, and sample image feature vector; if the determination result is no, then using another person in the sample image as the first person, or obtaining the next sample image.

[0007] Optionally, the weight calculation unit calculates the first weight and the second weight based on a self-attention mechanism.

[0008] Optionally, the weight calculation unit includes a face calculation subunit, a body calculation subunit, and a normalization layer. The face calculation subunit is used to calculate a first initial weight corresponding to the sample face feature vector based on the first sample feature vector. The body calculation subunit is used to calculate a second initial weight corresponding to the sample body feature vector. The normalization layer is used to normalize the first initial weight and the second initial weight to obtain the first weight and the second weight.

[0009] Optionally, the method further includes: inputting the image to be classified into a trained emotion recognition model to obtain the emotion category of the person in the image to be classified as output by the emotion recognition model.

[0010] Optionally, the emotion recognition model further includes a feature extraction module, which is used to obtain the sample facial feature vector, sample body feature vector, and sample image feature vector based on the sample image; wherein, if the number of sample images is less than a number threshold, the feature extraction module is in a frozen state during the process of the feature fusion module based on the loss.

[0011] Optionally, before obtaining the sample facial feature vector, sample body feature vector, and sample image feature vector based on the sample images, the method further includes: if the number of sample images is greater than or equal to a number threshold, then selecting the person with the best image quality in the facial region and the body region from the sample images as the first person.

[0012] This application embodiment also provides a training device for an emotion recognition model. The emotion recognition model includes a feature fusion module, which includes a weight calculation unit and a fusion unit. The device includes: a feature calculation module, used to obtain a sample facial feature vector, a sample body feature vector, and a sample image feature vector based on a sample image, wherein the sample facial feature vector is obtained by feature extraction of the facial region of a first person in the sample image, the sample body feature vector is obtained by feature extraction of the body region of the first person in the sample image, and the sample image feature vector is obtained by feature extraction of the sample image; and a weight calculation module, used to calculate the sample facial feature vector based on the first sample feature vector and the second sample feature vector using the weight calculation unit. The model consists of a first weight corresponding to the vector and a second weight corresponding to the sample body feature vector, wherein the first sample feature vector is obtained by fusing the sample facial feature vector and the sample image feature vector, and the second sample feature vector is obtained by fusing the sample body feature vector and the sample image feature vector; a fusion module is used to fuse the sample facial feature vector and the sample body feature vector according to the first weight and the second weight to obtain a third sample feature vector; a prediction module is used to calculate the prediction result of the emotion category of the first person based on the third sample feature vector; and an update module is used to calculate the loss according to the prediction result and the emotion category label of the first person, and update the feature fusion module according to the loss until the emotion recognition model converges.

[0013] This application also provides a computer-readable storage medium storing a computer program thereon, which, when run by a processor, executes the steps of the above-described training method for the emotion recognition model.

[0014] This application embodiment also provides a terminal, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the steps of the above-described emotion recognition model training method when running the computer program.

[0015] Compared with the prior art, the technical solution of this application embodiment has the following beneficial effects:

[0016] In this embodiment, the sample facial feature vector and sample image feature vector are first fused to obtain a first sample feature vector, and the sample body feature vector and sample image feature vector are fused to obtain a second sample feature vector. Then, a weight calculation unit calculates a first weight corresponding to the sample facial feature vector and a second weight corresponding to the sample body feature vector based on the first and second sample feature vectors. Further, a fusion unit fuses the sample facial feature vector and sample body feature vector according to the first and second weights to obtain a third sample feature vector. Finally, prediction and model updates are performed based on the third sample feature vector.

[0017] In the training process described above, global information (i.e., the feature vector of the sample image) participates in the calculation of weights, but does not participate in the feature fusion process based on the weights.

[0018] Specifically, on the one hand, the two effective local information (i.e., sample facial feature vector and sample body feature vector) are combined with global information to determine the weight of the local information. Compared with determining the weight of each local information based on only the two local information, the weight learning using the combined information can make the weight value calculated by the trained weight calculation unit more accurately represent the proportion of facial and body regions in emotional expression.

[0019] On the other hand, the two effective local information are fused according to the determined weights. That is, the global information only participates in the weight calculation process, but does not participate in the feature fusion process. By adopting such a scheme, we can ensure that effective feature information related to emotion expression is fused and learned, and reduce the interference of redundant information (such as the background in the sample image).

[0020] Therefore, the emotion recognition model trained using the training method provided in this application can accurately determine the weights of effective local information, and perform feature fusion and prediction based on the weights of effective local information, thereby accurately recognizing the emotions of people in images. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the architecture of an emotion recognition model in an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating a training method for an emotion recognition model according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the structure of a training device for an emotion recognition model according to an embodiment of this application. Detailed Implementation

[0024] As described in the background section, the accuracy of current emotion recognition methods needs further improvement.

[0025] Since facial expressions are one of the most expressive features of emotions, existing emotion recognition methods usually only recognize the facial regions in images. However, since the facial features captured in actual shooting scenarios may not be comprehensive, the emotion recognition results are not accurate enough.

[0026] In view of this, this application provides a training method for an emotion recognition model. In the scheme of this application embodiment, the trained emotion recognition model is based on the facial and body regions of a person in a sample image to accurately identify the person's emotions.

[0027] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] Reference Figure 1 , Figure 1 This is a schematic diagram of the architecture of an emotion recognition model in an embodiment of this application.

[0029] like Figure 1 As shown, the emotion recognition model 10 may include: a recognition module 11, a feature extraction module 12, a feature fusion module 13, and a prediction module 14.

[0030] It should be noted that during the model training phase, the images input to the emotion recognition model 10 are sample images, and the output of the emotion recognition model 10 is the predicted emotion category of the person in the sample image; during the model testing phase, the images input to the emotion recognition model 10 are test images, and the output of the emotion recognition model 10 is the predicted emotion category of the person in the test image; during the model inference phase, the images input to the emotion recognition model 10 are images to be classified, and the output of the emotion recognition model 10 is the predicted emotion category of the person in the images to be classified.

[0031] Specifically, the recognition module 11 is used to identify target regions in an image, which may include facial regions and body regions. The facial region refers to the area containing the image of a person's face in the image, and the body region refers to the area containing the image of a person's body in the image.

[0032] More specifically, the recognition module 11 may include a first recognition unit 111 and a second recognition unit 112, wherein the first recognition unit 111 may be used to recognize the facial regions of each person in the image, and the second recognition unit 112 may be used to recognize the body regions of each person in the image.

[0033] For example, the first recognition unit 111 can be a neural network unit with face detection function, such as a YOLO-face structure, and the second recognition unit can be a neural network unit with human body detection function, such as a YOLO structure.

[0034] Furthermore, the output of the recognition module 11 is connected to the input of the feature extraction module 12, which is used to extract features from at least a portion of the image.

[0035] In the embodiment of this application, the feature extraction module 12 is used to extract features from the facial region output by the recognition module 11 to obtain a facial feature vector. The feature extraction module 12 is also used to extract features from the body region output by the recognition module 11 to obtain a body feature vector. Furthermore, the feature extraction module is also used to extract features from the entire image to obtain an image feature vector. It should be noted that the image feature vector includes both facial and body feature vectors.

[0036] In the embodiments of this application, facial feature vectors and body feature vectors are two types of feature information related to emotion expression, which can be regarded as valid information. Other feature information in the image feature vectors other than facial feature vectors and body feature vectors can be regarded as redundant information, that is, invalid information.

[0037] Specifically, the feature extraction module 12 may include a first feature extraction unit 121, a second feature extraction unit 122, and a third feature extraction unit 123. The first feature extraction unit 121 can be used to extract features from the facial region, the second feature extraction unit 122 can be used to extract features from the body region, and the third feature extraction unit 123 can be used to extract features from the entire image.

[0038] For example, the first feature extraction unit 121, the second feature extraction unit 122 and the third feature extraction unit 123 can all be five-layer convolutional neural networks, but are not limited thereto.

[0039] More specifically, the process of extracting facial feature vectors can be represented as: F f =C1(I f ,θ f )

[0040] Among them, F f Let C1 represent the facial feature vector, and C1 represent the first feature extraction unit 121. The network parameters of the first feature extraction unit 121 are represented as θ. f I f This indicates the facial area.

[0041] The process of extracting body feature vectors can be represented as: Fb =C2(I b ,θ b )

[0042] Among them, F b C1 represents the body feature vector, C2 represents the second feature extraction unit 122, and the network parameters of the second feature extraction unit 122 are θ. b I b Indicates a body region.

[0043] The process of extracting image feature vectors can be represented as: F c =C3(I c ,θ c )

[0044] Among them, F c C3 represents the image feature vector, and C3 represents the third feature extraction unit 123. The network parameters of the third feature extraction unit 123 are θ. c I c Represents an image.

[0045] Therefore, the output of the feature extraction module 12 may include: facial feature vector, body feature vector and image feature vector.

[0046] Furthermore, the output of the feature extraction module 12 is connected to the input of the feature fusion module 13.

[0047] In the scheme of this application embodiment, the feature fusion module 13 may include a weight calculation unit 131 and a fusion unit 132. The weight calculation unit 131 is used to calculate the weights corresponding to the facial feature vector and the body feature vector, respectively. The fusion unit 132 is used to perform fusion processing on the facial feature vector and the body feature vector based on the weights corresponding to the facial feature vector and the body feature vector.

[0048] Specifically, the inputs to the weight calculation unit 131 are the facial feature vector, the body feature vector, and the image feature vector, and the outputs of the weight calculation unit 131 are the weight values ​​corresponding to the facial feature vector and the weight values ​​corresponding to the body feature vector.

[0049] In other words, the weight calculation unit 131 calculates the weight values ​​corresponding to the facial feature vector and the weight values ​​corresponding to the body feature vector based on the facial feature vector, the body feature vector and the image feature vector.

[0050] Specifically, the weight values ​​corresponding to facial feature vectors can be calculated based on facial feature vectors and image feature vectors, and the weight values ​​corresponding to body feature vectors can be calculated based on body feature vectors and image feature vectors.

[0051] As one possible implementation, the weight calculation unit 131 may include: a pooling layer, a face calculation subunit, a body calculation subunit, and a normalization layer.

[0052] In this system, the facial feature vector, body feature vector, and image feature vector output by the feature extraction module 12 are all multi-dimensional vectors. The pooling layer can be used to stretch these vectors into one-dimensional vectors. The facial calculation subunit can calculate the first initial weight corresponding to the facial feature vector based on the facial and image feature vectors. The body calculation subunit can calculate the second initial weight corresponding to the body feature vector based on the body and image feature vectors. The normalization layer can normalize the initial weight values ​​calculated by the facial and body calculation subunits to obtain the weight values ​​corresponding to the facial and body feature vectors.

[0053] Suppose that the one-dimensional vector corresponding to the facial feature vector can be denoted as V. f The one-dimensional vector corresponding to the body feature vector can be represented as V. b The one-dimensional vector corresponding to the image feature vector can be represented as V. c .

[0054] Specifically, the input to the facial calculation subunit can be represented as: The input to the body computation subunit can be represented as: Where [] indicates splicing, that is, It is composed of facial feature vector F f The corresponding one-dimensional vector V f and image feature vector F c The corresponding one-dimensional vector V c The concatenated vector It is composed of body feature vector F b The corresponding one-dimensional vector V b and image feature vector F c The corresponding one-dimensional vector V c The vector obtained by concatenation.

[0055] Furthermore, the face computation subunit can be used to calculate the first initial weights corresponding to the face feature vector based on the self-attention mechanism, and the body computation subunit can be used to calculate the second initial weights corresponding to the body feature vector based on the self-attention mechanism.

[0056] Specifically, the calculation process of the facial calculation subunit can be represented as follows: The calculation process of the body calculation subunit can be represented as follows:

[0057] Among them, Sf W1 represents the first initial weight corresponding to the facial feature vector, W1 represents the facial computation subunit, and the network parameters of W1 are represented as φ. f S b W2 represents the second initial weight corresponding to the body feature vector, W2 represents the body computation subunit, and the network parameters of W2 are represented as φ. b .

[0058] Furthermore, the normalization layer can be applied to S f and S b Perform normalization to obtain w f and w b Among them, w f w represents the weight value corresponding to the facial feature vector. b These are the weight values ​​corresponding to the body feature vectors.

[0059] Therefore, the weight values ​​output by the weight calculation unit 131 can be obtained, and the weight values ​​include the weight value w corresponding to the facial feature vector. f The weight value w corresponding to the body feature vector b w f +w b =1.

[0060] Furthermore, the input of the fusion unit 132 is connected to the output of the weight calculation unit 131, the output of the first feature extraction unit 121, and the output of the second feature extraction unit 122, respectively.

[0061] Specifically, the fusion unit 132 can be used based on the weight value w corresponding to the facial feature vector. f The weight value w corresponding to the body feature vector b The facial feature vector and body feature vector are fused to obtain a fused feature vector. The fusion process can be any existing suitable vector fusion method, and this embodiment is not limited to any particular method.

[0062] For example, the calculation process of the fusion unit 132 can be expressed as: V = [V f ⊙w f V b ⊙w b ], where V represents the fused feature vector and ⊙ represents the element-wise multiplication at corresponding positions.

[0063] Furthermore, the output of the fusion unit 132 is connected to the input of the prediction module 14. The prediction module 14 is used to calculate the predicted emotion category based on the fused feature vector output by the fusion unit 132. This embodiment does not limit the specific structure of the prediction module 14; the prediction module 14 can be any existing suitable classifier.

[0064] For example, the prediction module 14 can be a feedforward neural network structure. The prediction module 14 can include two fully connected layers and an activation function layer. The activation function layer is located between the two fully connected layers and can be configured with an activation function based on the Rectified Linear Unit (ReLU).

[0065] about Figure 1 For more information on the training process, working principle, and beneficial effects of the emotion recognition model shown below, please refer to the detailed description.

[0066] Reference Figure 2 , Figure 2 This is a flowchart illustrating a training method for an emotion recognition model according to an embodiment of this application. The specific details of the emotion recognition model can be found in [reference needed]. Figure 1 The method can be executed by a terminal, which can be any suitable device with data processing capabilities, such as a mobile phone, computer, IoT device, server, etc., but is not limited thereto.

[0067] Figure 2 The training method shown uses sample images from the training dataset for training, and these sample images have pre-labeled emotion categories. Specifically, the sample images may include one or more people, and for each person in the sample images, an emotion category label for that person may be pre-labeled. Figure 2 The method shown uses the first person as an example to describe the training method of the emotion recognition model.

[0068] For example, the sample image may be a preprocessed image, and the preprocessing includes one or more of image cropping, image flipping, image equalization, affine transformation and image enhancement, but is not limited thereto.

[0069] It should be noted that, Figure 2 This only illustrates the process of updating the model once based on a sample image during the training of the emotion recognition model. In practical applications, steps S21 to S25 can be repeated to ensure that the evaluation metric of the emotion recognition model reaches the set target value.

[0070] Figure 2 The training methods for the emotion recognition model shown may include:

[0071] Step S21: Based on the sample image, obtain the sample facial feature vector, the sample body feature vector, and the sample image feature vector, wherein the sample facial feature vector is obtained by extracting features from the facial region of the first person in the sample image, the sample body feature vector is obtained by extracting features from the body region of the first person in the sample image, and the sample image feature vector is obtained by extracting features from the sample image.

[0072] Step S22: Using the weight calculation unit, based on the first sample feature vector and the second sample feature vector, calculate the first weight corresponding to the sample facial feature vector and the second weight corresponding to the sample body feature vector, wherein the first sample feature vector is obtained by fusing the sample facial feature vector and the sample image feature vector, and the second sample feature vector is obtained by fusing the sample body feature vector and the sample image feature vector;

[0073] Step S23: Using a fusion unit, the sample facial feature vector and the sample body feature vector are fused according to the first weight and the second weight to obtain a third sample feature vector;

[0074] Step S24: Based on the third sample feature vector, calculate the prediction result of the emotion category of the first person;

[0075] Step S25: Calculate the loss based on the prediction result and the emotion category label of the first person, and update the feature fusion module based on the loss until the emotion recognition model converges.

[0076] It is understandable that, in specific implementation, the above method can be implemented using a software program that runs in the processor integrated inside the chip or chip module; or, the method can be implemented using hardware or a combination of hardware and software, such as using a dedicated chip or chip module, or using a dedicated chip or chip module combined with a software program.

[0077] Before step S21, the first person in the sample image can be determined.

[0078] As one possible implementation, the first person can refer to any person in the sample image. Specifically, the first person can be any person in the sample image who was not involved in the training.

[0079] As another possible implementation, the first person is the one with the best image quality among multiple people in the sample images. For example, if the number of sample images in the training dataset is greater than or equal to a threshold, the person with the best image quality in both the facial and body regions can be selected as the first person. It should be noted that the method for measuring image quality can be any existing suitable method, such as resolution, peak signal-to-noise ratio, etc., and this embodiment is not limited in this regard.

[0080] As another possible implementation, the ratio of the size of the face area to the body area of ​​the first character is less than a preset threshold.

[0081] Specifically, before step S21, the ratio of the size of the facial region to the size of the body region of the first person is calculated. If the ratio is greater than a preset threshold, it indicates that the facial region occupies a larger area in the sample image, while the body region occupies a smaller area. Normally, the facial region has a larger weight than the body region. However, in the above case, the first weight of the facial region obtained in step S22 may be too large, and the second weight of the body region may be too small. Such weight values ​​are based on area and may not be due to the features of emotional expression. Learning based on such weight values ​​will affect the accuracy of the weight calculation unit.

[0082] Therefore, in this embodiment, if the ratio of the size of the face region to the size of the body region of the first person is less than a preset threshold, step S21 can be continued. If the ratio is greater than or equal to the preset threshold, another person in the sample image can be used as the first person, or the next sample image can be obtained.

[0083] In step S21, the facial and body regions of the first person in the sample image can be identified first. Specifically, the sample images are input into... Figure 1 The first recognition unit 111 and the second recognition unit 112 are shown. The first recognition unit 111 can recognize the facial region, and the second recognition unit 112 can recognize the body region.

[0084] Furthermore, feature extraction can be performed on sample images, facial regions, and body regions.

[0085] Specifically, the sample facial region can be input into... Figure 1 The first feature extraction unit 121 shown obtains the sample facial feature vector output by the first feature extraction unit 121; and inputs the sample body region into... Figure 1The second feature extraction unit 122 shown obtains the sample body feature vector output by the second feature extraction unit 122; and outputs the sample image to the third feature extraction unit 123 to obtain the sample image feature vector output by the third feature extraction unit 123.

[0086] It should be noted that the sample image feature vector includes sample facial feature vectors and sample body feature vectors. In other words, the sample image feature vector can be a global feature vector of the image, while the sample facial feature vector and sample body feature vector are two different local feature vectors.

[0087] In step S22, the first weight corresponding to the sample facial feature vector and the second weight corresponding to the sample body feature vector are calculated.

[0088] Specifically, the facial feature vector and the image feature vector can be fused to obtain a first sample feature vector, and the body feature vector and the image feature vector can be fused to obtain a second sample feature vector. In specific implementations, the fusion process can be any existing appropriate vector fusion method, and this embodiment does not limit it.

[0089] Furthermore, based on the first sample feature vector, a first weight corresponding to the sample facial feature vector can be calculated, and based on the second sample feature vector, a second weight corresponding to the sample body feature vector can be calculated.

[0090] For example, a self-attention mechanism can be used to calculate a first weight based on the feature vector of the first sample, and a self-attention mechanism can be used to calculate a second weight based on the feature vector of the second sample.

[0091] Furthermore, the calculated first and second weights can be normalized. As mentioned above, the weight calculation unit may include a face calculation subunit, a body calculation subunit, and a normalization layer. In step S22, the first sample feature vector can be input into the face calculation subunit to obtain the first initial weight output by the face calculation subunit, and the second sample feature vector can be input into the body calculation subunit to obtain the second initial weight output by the body calculation subunit. Then, the first and second initial weights are normalized to obtain the first weight and the second weight.

[0092] For more information on step S22, please refer to the above text. Figure 1 The relevant description of the weight calculation unit 131 in the text will not be repeated here.

[0093] In step S23, the sample facial feature vector and sample body feature vector can be fused based on the first weight and the second weight to obtain the third sample feature vector.

[0094] Specifically, the first weight and the second weight can be input to the sample's facial feature vector and sample's body feature vector. Figure 1 The fusion unit 132 shown is used to obtain the third sample feature vector output by the fusion unit 132. That is, the third sample feature vector is obtained by fusing the sample facial feature vector and the sample body feature vector according to the first weight and the second weight.

[0095] For more information on step S23, please refer to the above text. Figure 1 The relevant description of the fusion unit 132 in the text will not be repeated here.

[0096] In step S24, the predicted emotion category of the first person can be calculated based on the feature vector of the third sample.

[0097] Specifically, the feature vector of the third sample can be input into... Figure 1 The prediction module 14 shown calculates the emotion category of the first person based on the feature vector of the third sample.

[0098] In step S25, the loss can be calculated based on the prediction results obtained in S24 and the emotion category label of the first person, and the model can be updated according to the calculated loss.

[0099] For example, the loss may be cross-entropy loss, or the loss may be calculated based on other existing loss functions; this embodiment is not limited to this.

[0100] Furthermore, the model can be updated based on the loss. Specifically, in step S25, the feature fusion module and the prediction module can be updated based on the loss. In addition, the feature extraction module can also be updated based on the loss. Furthermore, the feature recognition module can also be updated based on the loss.

[0101] Figure 2 The training method shown can be applied to the initial training phase of an emotion recognition model, or... Figure 2 The method shown can also be applied to the optimization phase of emotion recognition models.

[0102] In one embodiment of this application, Figure 2 The training shown is the training during the model optimization process, and the modules that need to be updated can be determined based on the number of sample images used for model optimization.

[0103] Specifically, if the number of sample images is less than the threshold, the feature fusion module and prediction module can be updated based on the loss, but the feature extraction module and feature recognition module can be frozen. If the number of sample images is greater than or equal to the threshold, the modules in the emotion recognition model can be updated based on the loss.

[0104] Here, the "frozen state" refers to the network parameters remaining unchanged, and model optimization can refer to fine-tuning an already trained model. The threshold for the number of samples can be preset. When the number of sample images is limited, updating each module would prevent the model from converging quickly. Therefore, the above scheme is beneficial for quickly achieving model fine-tuning when the number of sample images is limited.

[0105] In another embodiment of this application, Figure 2 The training shown is the initial training process of the model, in which the individual modules of the emotion recognition model can be updated based on the loss.

[0106] The specific method for updating based on loss can refer to existing methods for adjusting model parameters based on loss, such as gradient descent, etc. This embodiment does not limit this.

[0107] In practical implementation, the emotion recognition model can also be tested. Testing can determine the evaluation metrics for the emotion recognition model. These metrics can be any one or more of the following: recall, precision, accuracy, and the F1-score, but are not limited to these.

[0108] The recall rate is calculated as follows: R = t p / (t p +f n Precision is calculated as follows: P = t p / (t p +f p The accuracy rate is calculated as follows: A = (t) p +t n ) / (t p +t n +f n +f p The comprehensive evaluation index F1-score is calculated as follows: F1 = 2 × P × R / (P + R).

[0109] Where R represents recall, P represents precision, A represents accuracy, F1 represents F1-score, and t p t represents the number of positive classes predicted as positive. nf represents the number of negative classes predicted as negative. n f represents the number of underreported cases. p This indicates the number of false alarms.

[0110] If the evaluation metric of the emotion recognition model exceeds the set target value, the emotion recognition model can be used for emotion recognition. In the embodiments of this application, "model convergence" can refer to the model's evaluation metric exceeding the set target value.

[0111] During the model inference phase, the image to be classified can be input into the trained emotion recognition model to obtain the emotion category of the person in the image, output by the emotion recognition model. If the image to be classified includes multiple people, the emotion recognition model can calculate the emotion category of each person separately. For the specific process of calculating the emotion category of each person in the image using the emotion recognition model during the model inference phase, please refer to the section above. Figure 1 and Figure 2 The relevant descriptions will not be repeated here.

[0112] Reference Figure 3 , Figure 3 This is a schematic diagram of the structure of a training device for an emotion category model according to an embodiment of this application. Figure 3 As shown, Figure 3 The apparatus shown may include:

[0113] The feature calculation module 31 is used to obtain a sample facial feature vector, a sample body feature vector, and a sample image feature vector based on the sample image. The sample facial feature vector is obtained by extracting features from the facial region of the first person in the sample image, the sample body feature vector is obtained by extracting features from the body region of the first person in the sample image, and the sample image feature vector is obtained by extracting features from the sample image.

[0114] The weight calculation module 32 is used to calculate, using the weight calculation unit, a first weight corresponding to the sample facial feature vector and a second weight corresponding to the sample body feature vector based on the first sample feature vector and the second sample feature vector, wherein the first sample feature vector is obtained by fusing the sample facial feature vector and the sample image feature vector, and the second sample feature vector is obtained by fusing the sample body feature vector and the sample image feature vector;

[0115] The fusion module 33 is used to fuse the sample facial feature vector and the sample body feature vector according to the first weight and the second weight to obtain a third sample feature vector.

[0116] The prediction module 34 is used to calculate the prediction result of the emotion category of the first person based on the feature vector of the third sample.

[0117] The update module 35 is used to calculate the loss based on the prediction result and the emotion category label of the first person, and update the feature fusion module based on the loss until the emotion recognition model converges.

[0118] For more information on the working principle, working method, and beneficial effects of the training device for the emotion category model in this application embodiment, please refer to the above description of the training method for the emotion category model, which will not be repeated here.

[0119] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it performs the steps of the above-described training method for the emotion category model. The storage medium may include ROM, RAM, a disk, or an optical disk, etc. The storage medium may also include non-volatile memory or non-transitory memory, etc.

[0120] This application embodiment also provides a terminal, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of the above-described training method for the emotion category model. The terminal includes, but is not limited to, terminal devices such as mobile phones, computers, and tablet computers.

[0121] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0122] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0123] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or in a combination of hardware and software functional units. For example, for various devices or products applied to or integrated into a chip, each module / unit can be implemented using hardware such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining (if any) modules / units can be implemented using hardware such as circuits; for various devices or products applied to or integrated into a chip module, each module / unit can be implemented using hardware such as circuits, and different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or different components of the chip module, or at least some modules / units can be implemented using hardware such as circuits. The components can be implemented using software programs that run on the processor integrated within the chip module. The remaining (if any) modules / units can be implemented using hardware methods such as circuits. For various devices and products applied to or integrated into the terminal, each of its components / units can be implemented using hardware methods such as circuits. Different modules / units can be located in the same component (e.g., chip, circuit module, etc.) or in different components within the terminal. Alternatively, at least some modules / units can be implemented using software programs that run on the processor integrated within the terminal, while the remaining (if any) modules / units can be implemented using hardware methods such as circuits.

[0126] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article indicates that the preceding and following related objects have an "or" relationship.

[0127] In this application's embodiments, "multiple" refers to two or more. The descriptions of "first," "second," etc., appearing in this application's embodiments are merely illustrative and for distinguishing the described objects; they have no order and do not indicate a specific limitation on the number of devices in this application's embodiments, nor do they constitute any limitation on the embodiments of this application.

[0128] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application shall be determined by the scope defined in the claims.

Claims

1. A training method for an emotion recognition model, characterized in that, The emotion recognition model includes a feature fusion module, which includes a weight calculation unit and a fusion unit. The method includes: Based on the sample image, sample facial feature vector, sample body feature vector, and sample image feature vector are obtained. The sample image feature vector includes sample facial feature vector and sample body feature vector. The sample facial feature vector is obtained by extracting features from the facial region of the first person in the sample image. The sample body feature vector is obtained by extracting features from the body region of the first person in the sample image. The sample image feature vector is obtained by extracting features from the entire sample image. The sample image feature vector is a global feature vector of the image. The sample facial feature vector and the sample body feature vector are two different local feature vectors. Using the weight calculation unit, based on the first sample feature vector and the second sample feature vector, the first weight corresponding to the sample facial feature vector and the second weight corresponding to the sample body feature vector are calculated, wherein the first sample feature vector is obtained by fusing the sample facial feature vector and the sample image feature vector, and the second sample feature vector is obtained by fusing the sample body feature vector and the sample image feature vector; A fusion unit is used to fuse the sample facial feature vector and the sample body feature vector according to the first weight and the second weight to obtain a third sample feature vector; Based on the feature vector of the third sample, the predicted result of the emotion category of the first person is calculated; The loss is calculated based on the prediction result and the emotion category label of the first person, and the feature fusion module is updated based on the loss until the emotion recognition model converges.

2. The method of claim 1, wherein, Before obtaining the sample facial feature vector, sample body feature vector, and sample image feature vector based on the sample image, the method further includes: Calculate the ratio of the size of the facial region to the size of the body region of the first person; Determine whether the ratio is less than a preset threshold. If the determination result is yes, then obtain the sample facial feature vector, sample body feature vector, and sample image feature vector. If the determination result is no, then take another person in the sample image as the first person, or obtain the next sample image.

3. The method of claim 1, wherein, The weight calculation unit calculates the first weight and the second weight based on the self-attention mechanism.

4. The method of claim 3, wherein, The weight calculation unit includes a face calculation subunit, a body calculation subunit, and a normalization layer. The face calculation subunit is used to calculate the first initial weight corresponding to the sample face feature vector based on the first sample feature vector. The body calculation subunit is used to calculate the second initial weight corresponding to the sample body feature vector. The normalization layer is used to normalize the first initial weight and the second initial weight to obtain the first weight and the second weight.

5. The method of claim 1, wherein, The method further includes: The image to be classified is input into the trained emotion recognition model to obtain the emotion category of the person in the image to be classified, as output by the emotion recognition model.

6. The method of claim 1, wherein, The emotion recognition model further includes a feature extraction module, which is used to obtain the sample facial feature vector, the sample body feature vector, and the sample image feature vector based on the sample image. If the number of sample images is less than a threshold, the feature extraction module is frozen during the feature fusion process based on the loss.

7. The method of claim 1, wherein, Before obtaining the sample facial feature vector, sample body feature vector, and sample image feature vector based on the sample image, the method further includes: If the number of sample images is greater than or equal to a threshold, then the person with the best image quality in both the facial and body regions is selected from the sample images as the first person. 8.A device for training an emotion recognition model, comprising: The emotion recognition model includes a feature fusion module, which includes a weight calculation unit and a fusion unit. The device includes: The feature calculation module is used to obtain sample facial feature vectors, sample body feature vectors, and sample image feature vectors based on sample images. The sample image feature vectors include sample facial feature vectors and sample body feature vectors. The sample facial feature vectors are obtained by extracting features from the facial region of the first person in the sample image. The sample body feature vectors are obtained by extracting features from the body region of the first person in the sample image. The sample image feature vectors are obtained by extracting features from the entire sample image. Furthermore, the sample image feature vectors are global feature vectors of the image, while the sample facial feature vectors and sample body feature vectors are two different local feature vectors. The weight calculation module is used to calculate, based on the first sample feature vector and the second sample feature vector, a first weight corresponding to the sample facial feature vector and a second weight corresponding to the sample body feature vector using the weight calculation unit, wherein the first sample feature vector is obtained by fusing the sample facial feature vector and the sample image feature vector, and the second sample feature vector is obtained by fusing the sample body feature vector and the sample image feature vector; The fusion module is used to fuse the sample facial feature vector and the sample body feature vector according to the first weight and the second weight to obtain a third sample feature vector. The prediction module is used to calculate the prediction result of the emotion category of the first person based on the feature vector of the third sample. An update module is used to calculate the loss based on the prediction result and the emotion category label of the first person, and update the feature fusion module based on the loss until the emotion recognition model converges.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is run by the processor, it performs the steps of the training method for the emotion recognition model according to any one of claims 1 to 7.

10. A terminal comprising a memory and a processor, said memory having stored thereon a computer program that is operable on said processor, characterized in that, When the processor runs the computer program, it performs the steps of the training method for the emotion recognition model according to any one of claims 1 to 7.

Images