Method, device, electronic device and storage medium for determining emotion of facial image

Abstract

The present invention provides a method, device, electronic device and storage medium for determining emotions of facial images, wherein the method comprises: performing convolution calculation on a facial image to be analyzed to extract full-image features of the facial image to be analyzed; decoupling the full-image features to obtain features of multiple facial components; based on a dynamic fusion basis state, dynamically fusing the features of the multiple facial components to form high-level semantic features; based on the high-level semantic features and a language generation model, generating an emotion description sentence in a word-by-word prediction manner, thereby improving the quality of the generated emotion description sentence.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to a method, device, electronic equipment and storage medium for determining emotions of facial images. Background Art

[0002] Emotional analysis technology enables the understanding and characterization of facial expressions and the emotions they reflect. At present, emotional analysis technology has a wide range of application needs in the fields of smart campuses, public security control, etc. For example, through emotional analysis, the education department can understand the emotional fluctuations of students, and make early detection, early investigation, and early intervention of possible psychological abnormalities, helping to reduce campus safety risks and students' psychological pressure, and providing a basis for schools and education departments to make scientific and accurate decisions.

[0003] Existing sentiment analysis technology is mainly implemented by label classification. The label classification method divides emotions into several categories, which is contrary to the rich and diverse characteristics of facial expressions, resulting in large intra-class variance and high inter-class confusion.

[0004] Existing image description technologies are not suitable for processing unstructured facial images. In order to achieve fine-grained and interpretable language generation, existing image description technologies need to structure images based on detection technology, then reason about the relationships between instances and finally generate language. However, for a large number of unstructured facial images in actual scenes, there is a lack of corresponding detection technology to abstract the features of their components, which limits the application of image description technology in the field of facial emotions. Existing image description technologies have limitations such as weak interpretability and limited description performance. Summary of the invention

[0005] The present invention provides a method, device, electronic device and storage medium for determining emotions of facial images, so as to solve the defects of poor interpretability and limited description performance in emotion recognition of unstructured facial images in the prior art, and improve the quality of generated emotion description sentences.

[0006] The present invention provides a method for determining emotion of a facial image, comprising:

[0007] Perform convolution calculation on the face image to be analyzed to extract full-image features of the face image to be analyzed;

[0008] Decoupling the full-image features to obtain features of multiple facial parts;

[0009] Based on the dynamic fusion basis state, dynamically fuse the features of the multiple facial parts to form a high-level semantic feature;

[0010] Based on the high-level semantic features and the language generation model, emotion description sentences are generated in a word-by-word prediction manner.

[0011] According to the method for determining the emotion of a facial image provided by the present invention, the method dynamically fuses the features of the plurality of facial components based on the dynamic fusion state to form a high-level semantic feature, including:

[0012] Initialize dynamic fusion based on state;

[0013] Initialize the first word of the emotion description sentence to be an empty tag;

[0014] Based on the current dynamic fusion basis state, a soft attention mechanism is used to fuse the features of the multiple face parts to obtain the t-th high-level semantic feature;

[0015] The generating of emotion description sentences in a word-by-word prediction manner based on the high-level semantic features and the language generation model includes:

[0016] Input the t-th high-level semantic feature into the language generation model, generate the t-th word, update the dynamic fusion basis state, perform an addition operation on t, and start generating the next word until all the words included in the emotion description sentence are generated;

[0017] All the words generated by the language generation model are concatenated in order to obtain the emotion description sentence;

[0018] Wherein, t is greater than or equal to 1 and less than or equal to the total number of words contained in the emotion description sentence.

[0019] According to the method for determining emotion of a facial image provided by the present invention, the decoupling of the full image features to obtain features of multiple facial parts includes:

[0020] Performing a convolution transformation on the full-image feature to obtain a first mask group including a plurality of facial part masks;

[0021] Performing a dimensionality reduction operation on the full-image features to obtain a feature map after dimensionality reduction;

[0022] Expanding the channel dimensions of the plurality of facial part masks in the first mask group in a repeated manner to obtain a plurality of expanded facial part masks, wherein the channel dimensions of the expanded facial part masks are the same as the channel dimensions of the feature map after dimensionality reduction;

[0023] Applying the multiple expanded facial component masks to the feature map after dimensionality reduction to obtain multiple semantic feature maps corresponding to different facial components;

[0024] The multiple semantic feature maps are flattened into vectors, and subjected to feature transformation to obtain features of multiple facial parts of the facial image to be analyzed.

[0025] The method for determining emotion of a facial image provided by the present invention further includes:

[0026] The target model is trained according to the face image samples, the supervised annotation sentences corresponding to the face image samples and the predefined optimization objective function, and the parameters of the target model when the training is completed are saved to obtain the language generation model.

[0027] According to the method for determining emotions of facial images provided by the present invention, the target model is trained according to the training set images, the supervised annotation sentences corresponding to the training set images, and the predefined optimization objective function, and the parameters of the target model are saved when the training is completed, and the language generation model is obtained, which includes:

[0028] Obtaining a face image sample and a supervised annotation sentence corresponding to the face image sample;

[0029] Performing convolution calculation on the face image sample to extract full-image features of the face image sample;

[0030] Performing a convolution transformation on the full image features to obtain a second mask group including multiple facial part masks;

[0031] Calculating similarity scores between a plurality of facial part masks in the second mask group;

[0032] Based on the target model, predict the posterior distribution of the tth word;

[0033] Calculate the target distribution of the tth word of the supervised annotation sentence based on the semantic relevance;

[0034] Calculating a distribution difference score of the t-th word according to the posterior distribution of the t-th word and the target distribution of the t-th word;

[0035] Perform an addition operation on t and start the distribution difference score calculation process for the next word until the distribution difference scores of all words are obtained;

[0036] Obtaining an optimization target according to the distribution difference scores of all words, the similarity score and a second predefined hyperparameter;

[0037] Adjusting the parameters in the target model to minimize the value of the optimization target;

[0038] The adjusted parameters are saved to obtain the language generation model.

[0039] According to the method for determining emotions of facial images provided by the present invention, the target distribution of the t-th word of the supervised annotation sentence is calculated based on semantic relevance, including:

[0040] For the t-th word of the supervised labeled sentence, the semantic relevance between the t-th word and all words in the pre-stored candidate word list is calculated, and the semantic relevance is normalized to obtain a similarity distribution corresponding to the t-th word;

[0041] Based on the similarity distribution corresponding to the t-th word, the impulse distribution corresponding to the t-th word and a first predefined hyperparameter, a target distribution of the t-th word is calculated.

[0042] According to the method for determining emotions of facial images provided by the present invention, the optimization target is obtained according to the distribution difference scores of all words, the similarity scores and the second predefined hyperparameter, including:

[0043] According to the distribution difference scores of all words, the similarity score and the second predefined hyperparameter, an optimization target is calculated using an optimization target function;

[0044] Wherein, the optimization objective function is:

[0045] L=∑ t S t +λ×S m

[0046] Among them, S t is the difference score of the tth word, λ is the second predefined hyperparameter, S m Score the familiarity.

[0047] The present invention also provides a device for determining emotion of a facial image, comprising:

[0048] A face convolution module, used to perform convolution calculation on the face image to be analyzed and extract the full-image features of the face image to be analyzed;

[0049] A facial component decoupling module, used for decoupling the full image features to obtain features of multiple facial components;

[0050] A component relationship reasoning module, used for dynamically fusing the features of the plurality of face components based on a dynamic fusion basis state to form a high-level semantic feature;

[0051] The language generation module is used to generate emotion description sentences in a word-by-word prediction manner based on the high-level semantic features and the language generation model.

[0052] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for determining the emotion of a facial image when executing the program.

[0053] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for determining the emotion of a facial image are implemented.

[0054] The method, device, electronic device and storage medium for determining emotions in facial images provided by the present invention decouple the extracted full-image features and then dynamically fuse multiple facial component features obtained by decoupling to generate emotion description sentences in a word-by-word prediction manner, thereby improving the quality of the generated emotion description sentences. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of the flow of the method for determining the emotion of a facial image provided by the present invention;

[0057] Figure 2 A schematic diagram of a process for decoupling the full-image features to obtain features of multiple facial parts provided by the present invention;

[0058] Figure 3 A schematic diagram of a network structure for realizing decoupling of the full-image features provided by the present invention;

[0059] Figure 4 The mask provided by the present invention acts on the reduced-dimensional full-face feature map to obtain a feature map enhanced at each component;

[0060] Figure 5 A schematic diagram of the language generation model obtained by training according to the loss function is provided for the present invention;

[0061] Figure 6 It is a schematic diagram of the structure of the device for determining the emotion of a facial image provided by the present invention;

[0062] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Figure 1 A flow chart of a method for determining emotion of a facial image provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the following steps are included:

[0065] Step 100: Perform convolution calculation on the face image to be analyzed to extract full-image features of the face image to be analyzed.

[0066] Specifically, the present invention is applicable to determining emotions of unstructured facial images, for example, obtaining unstructured facial images of students through cameras at the school gate and storing them in a pool to be analyzed. Selecting facial images to be analyzed from the sample pool to be analyzed for convolution calculation. The sample pool to be analyzed is a collection of all facial images to be analyzed for emotions.

[0067] Due to computing power limitations, only a portion of the image samples in the sample pool to be analyzed can be analyzed in each batch of analysis and calculation, and all face images are processed in batches. The selected strategy can be determined according to project needs, such as first-in-first-out, first-in-last-out, etc., and is not specifically limited here.

[0068] A convolution calculation is performed on the face image to be analyzed selected from the sample pool to be analyzed to obtain a multi-channel feature map as the full-image feature of the face image to be analyzed. Each position of the multi-channel feature map corresponds to the corresponding area of ​​the face image to be analyzed. Therefore, the multi-channel feature map is a semantic representation of the full-image features of the face.

[0069] Step 101: decouple the full image features to obtain features of multiple facial parts.

[0070] Specifically, the full-image features obtained in step 100 are decoupled and separated into a group of features, which form a component feature group {F}, where each feature in the group represents a certain part of the face (such as mouth, nose, cheekbone, forehead, etc.). The purpose of this is to imitate the human cognitive process and improve the interpretability and generalization of the generated description sentences.

[0071] Step 102: Based on the dynamic fusion basis state, dynamically fuse the features of the multiple facial parts to form a high-level semantic feature.

[0072] Among them, the dynamic fusion based state is a state that is updated with the generation progress of the emotion description sentence. When the generation process has not yet started, the state needs to be initialized first. During the emotion description sentence generation process in step 103, the state is updated according to the generated text content.

[0073] Specifically, since the visual information required in each step of generating words in the process of generating sentences in step 103 in a word-by-word prediction manner is not completely consistent, it is necessary to dynamically fuse the features of multiple facial parts. During the dynamic fusion process, based on the dynamic fusion state, some features in the component feature group decoupled in step 101 can be emphasized, and a group of features can be fused into a high-level semantic feature as a visual representation of the image semantic information.

[0074] Step 103: Generate a sentiment description sentence in a word-by-word prediction manner based on the high-level semantic features.

[0075] Specifically, since the emotion description sentence is generated in a word-by-word prediction manner, it is necessary to continuously generate the next word in the sentence based on the high-level semantic features formed by dynamically fusing the features of the multiple facial parts in step 102, that is, continuously looping steps 102 and 103, generating a word each time through the loop, and updating the dynamic fusion based on the state, and the generated multiple words constitute the final emotion description sentence.

[0076] The embodiment of the present invention decouples the full-image features of the facial image to be analyzed, obtains features that respectively characterize each facial component, and then dynamically fuses multiple features, generates emotion description sentences using a word-by-word prediction method, thereby realizing emotion description of unstructured facial images, effectively refining the granularity of emotion analysis, and improving the quality of the generated emotion description sentences.

[0077] Figure 2 A schematic diagram of a process for decoupling the full image features to obtain features of multiple facial parts provided by an embodiment of the present invention, Figure 3 A schematic diagram of a network structure for implementing decoupling of the full-image features provided by an embodiment of the present invention. Figure 2 and Figure 3 In one embodiment, decoupling the full image features to obtain features of multiple facial parts includes the following sub-steps:

[0078] Step 200: Perform a convolution transformation on the full image features to obtain a first mask group including multiple facial part masks.

[0079] Specifically, the global feature F0 of the unstructured face image in step 100 is obtained. The dimension of the global feature F0 is w×h×N dimensions, where N is the channel dimension. Since the global feature F0 contains the semantic information of the face, a series of heatmaps (i.e., masks) corresponding to different face component positions can be obtained based on this semantic information through transformation. The global feature F0 is subjected to a convolutional transformation to obtain K masks that respond to different face components. Each mask obtains a response at a specific face component position, and the dimension of each mask is w×h. The set of K masks forms the first mask group {M}. Among them, the number K of face components can be predefined and adjusted according to actual needs.

[0080] Step 201: Perform a dimensionality reduction operation on the global feature to obtain a feature map after dimensionality reduction.

[0081] Specifically, a dimensionality reduction operation is performed on the global feature F0 to obtain a feature map F1 with a dimension of w×h×N’, ensuring that N’ < N to reduce the computational cost of subsequent steps.

[0082] Step 202: Expand the channel dimensions of multiple face component masks in the first mask group in a repetitive manner to correspondingly obtain multiple expanded face component masks, where the channel dimension of the expanded face component masks is the same as the channel dimension of the feature map after dimensionality reduction.

[0083] Specifically, the channel dimensions of the K masks in the first mask group {M} obtained in step 200 are expanded in a repetitive manner from w×h dimensions to w×h×N’ dimensions. The significance of dimensionality expansion is to give a unified corresponding intensity correction coefficient to the semantic representations of each channel at a certain spatial position.

[0084] Step 203: Apply the multiple expanded face component masks to the feature map after dimensionality reduction to correspondingly obtain multiple semantic feature maps that respond to different face components.

[0085] Specifically, Figure 4 The mask provided in the embodiment of the present invention is applied to the feature map of the entire face after dimensionality reduction to obtain a feature map enhanced at each component. As Figure 4 shown, the expanded i-th mask M i is applied to the feature map F1 obtained by dimensionality reduction in step 201 to obtain a semantic feature map F2 corresponding to the face component corresponding to the mask. i, the mode of action is element-wise multiplication (Hadamard product). The K expanded masks in step 202 act on the feature map F1 respectively to obtain K semantic feature maps, which form a feature map group {F2}, and the dimension of each feature map is w×h×N'. Since each mask only obtains a response at the corresponding face part, this step filters out the features of the non-current part position, realizing the separation of the current part feature from the full face image feature.

[0086] Step 204: Flatten the multiple semantic feature maps into vectors, and perform feature transformation to obtain features of each facial component of the facial image to be analyzed.

[0087] Specifically, the flattening operation can ensure that the spatial position information and deformation information, which are very important for describing emotions, are retained, but it will inevitably bring dimensional redundancy. Feature transformation can achieve dimensionality reduction on this basis, obtain more compact component features, and reduce the number of parameters in subsequent steps.

[0088] The embodiment of the present invention decouples the full-image features of the face image to be analyzed, obtains features that characterize each face component, and then dynamically fuses multiple features to generate an emotion description sentence using a word-by-word prediction method, thereby achieving an emotion description of unstructured face images, effectively refining the granularity of emotion analysis, and improving the quality of the generated emotion description sentence. In the process of decoupling the full-image features, the face component mask is used to filter out features of non-current component positions, thereby achieving the extraction of current component features from the full-image features of the face, and then the semantic features are flattened and feature transformed to obtain more compact component features, thereby reducing the number of parameters in subsequent steps.

[0089] In one embodiment, based on the dynamic fusion basis state, dynamically fusing the features of the plurality of face parts to form a high-level semantic feature, and generating an emotion description sentence in a word-by-word prediction manner based on the high-level semantic feature and a language generation model includes the following sub-steps:

[0090] Step 300: Initialize the dynamic fusion based state.

[0091] Specifically, the dynamic fusion basis state is a state that is updated along with the generation progress of the emotion description sentence. When the generation process has not yet started, the state needs to be initialized first.

[0092] Step 301: Initialize the first word of the emotion description sentence to be an empty tag.

[0093] Specifically, since sentences are generated step by step in a word-by-word prediction manner, the current output word needs to be determined based on the generated words and visual information in each step of word generation. Therefore, when generating the first word of a sentence, it is necessary to provide an empty token as a preceding word for the language generation module to serve as a placeholder.

[0094] Step 302: Based on the current dynamic fusion basis state, a soft attention mechanism is used to fuse the features of the multiple facial parts to obtain the t-th high-level semantic feature.

[0095] Specifically, when performing feature fusion, it is necessary to determine the importance of the features of each facial component at the current moment based on the current dynamic fusion basis state, and use the soft attention mechanism to fuse the features of facial components. The fusion process can be expressed by the following formula:

[0096]

[0097]

[0098] Among them, F i is the ith feature in the component feature group {F}, h t is the fusion basis state at time t, This means that when generating the t-th word, the importance weight of the i-th feature satisfies F t The visual feature finally fused is also the t-th high-level semantic feature, which serves as the input basis for language generation at time t. So far, the t-th high-level semantic feature is obtained.

[0099] Step 303: input the t-th high-level semantic feature into the language generation model, generate the t-th word, update the dynamic fusion basis state, perform a plus 1 operation on t, and start generating the next word until all the words contained in the emotion description sentence are generated.

[0100] Among them, the language generation model is obtained by training the target model. By inputting the t-th high-level semantic feature into the language generation model, the t-th word of the emotion description sentence can be obtained.

[0101] After generating the tth word in the emotion description sentence, the dynamic fusion basis state needs to be updated to prepare for the generation of the t+1th high-level semantic feature. The dynamic fusion basis state is updated based on the sequence information of the generated words, the full-image features and the current dynamic fusion basis state. The update process can be expressed by the following formula:

[0102] h t =Update(F0,w ＜t ,h ＜t )

[0103] Among them, F0 is the whole image feature, w ＜t is the sequence information of the generated words, h ＜t Based on the previous dynamic fusion state.

[0104] After the dynamic fusion basis state is updated, t is incremented by 1 to start generating the next word, i.e., steps 302 and 303 are repeated until all the words contained in the emotion description sentence are generated. <eos>As the termination condition of the loop process. When this flag is output, it means that the complete and self-consistent description statement generation work has been completed, and the above loop process is terminated.

[0105] Step 304: concatenate all the words generated by the language generation model in order to obtain the emotion description sentence.

[0106] Specifically, all words generated by the language generation model are concatenated in the order of generation to obtain the emotion description sentence.

[0107] The embodiment of the present invention decouples the full-image features of the face image to be analyzed, obtains features that respectively characterize each facial component, and then dynamically fuses multiple features. Based on the high-level semantic features obtained by the fusion, an emotion description sentence is generated by adopting a word-by-word prediction method, thereby realizing the emotion description of unstructured face images, effectively refining the granularity of emotion analysis, and improving the quality of the generated emotion description sentences.

[0108] In one embodiment, it also includes: training the target model according to the face image samples, the supervised annotation sentences corresponding to the face image samples and the predefined optimization objective function, saving the parameters of the target model when the training is completed, and obtaining the language generation model.

[0109] Specifically, a target distribution can be obtained based on the supervised annotation sentences corresponding to the face image samples. During the training process, the processing results of the target model on the face image samples are fit to the target distribution, and the model parameters are continuously adjusted. The parameters of the target model after training are saved to obtain the language generation model. The training is completed based on the value of the predefined optimization objective function being less than the preset value, at which time the fitting process is considered to be completed.

[0110] The embodiment of the present invention obtains a language generation model through training based on face image samples, supervised annotation sentences corresponding to the face image samples and a predefined optimization objective function, thereby improving the accuracy of model recognition and the quality of the generated emotion description sentences.

[0111] In the traditional description task training method, the description process can be broken down into a series of word predictions. Each word can be regarded as a multi-classification problem when predicting (that is, each word is a category, and there are a total of vocabulary size categories). Therefore, the cross entropy loss of the classification problem can be used as the training loss function in the single word prediction process. At the sentence level, the cross entropy loss of each word that constitutes the sentence is added up as the final loss function. The form of the cross entropy loss function is:

[0112] L(θ)=-log p θ (c t |F t ,c ＜t )

[0113] Among them, c t Indicates the marking word, c ＜t Indicates the predicate, p θ () represents the posterior distribution of each word in the vocabulary modeled by the model under the condition of given image visual information and previous words. L(θ) can be written as:

[0114] L(θ)=-δ(w=c t )log p θ (w|F t ,c ＜t )

[0115] =δ(w=c t )logδ(w=c t )-δ(w=c t )log p θ (w|F t ,c ＜t )

[0116] Among them, δ(w=c t ) means that when w=c t The impulse distribution at . It can be seen that the cross entropy loss function is actually the impulse distribution δ(w=c t ) and the posterior distribution p of the model prediction θ During training, by optimizing the cross entropy loss function, the distance between the model's predicted posterior distribution and the impulse distribution of the labeled words can be reduced, thereby learning language generation. This approach has two problems:

[0117] Problem 1: It does not introduce any constraints on the facial component decoupling module. Since the training uses unstructured face images (i.e., no component mask reference annotations), the facial component decoupling module has no additional supervisory information to guide the parameter update process. Simply using the cross entropy loss function for training will cause the K masks predicted by the decoupling module to converge, resulting in decoupling failure.

[0118] Problem 2: The cross entropy loss function implies the assumption that all categories are mutually exclusive. However, there are a lot of synonyms used in emotion descriptions (such as joy and happiness, sadness and sadness, etc.), which may cause similar visual semantic information to be divided into different categories in different annotations. When the amount of data is small, samples with similar visual semantic information may be divided into multiple categories, resulting in a small number of training samples for each class, which brings about convergence problems.

[0119] In response to the above two questions, Figure 5 The network structure diagram of the language generation model obtained by training according to the loss function is provided for the embodiment of the present invention. Figure 5 As shown, in one embodiment, training the target model according to the face image samples, the supervised annotation sentences corresponding to the face image samples and the predefined optimization objective function, saving the parameters of the target model when the training is completed, and obtaining the language generation model includes the following steps:

[0120] Step 500: Obtain a face image sample and a supervised annotation sentence corresponding to the face image sample.

[0121] Specifically, a plurality of face image samples are obtained, and emotions of the face image samples are described by manual annotation, and the manually annotated emotion description sentences are used as supervised annotation sentences corresponding to the face image samples.

[0122] Step 501: Perform convolution calculation on the face image sample to extract full-image features of the face image sample.

[0123] Specifically, a convolution calculation is performed on the face image sample to obtain a multi-channel feature map as the full-image feature of the face image sample, and each position of the multi-channel feature map corresponds to a corresponding area of ​​the face image sample.

[0124] Step 502: Perform a convolution transformation on the full image features to obtain a second mask group including multiple facial part masks.

[0125] Specifically, the full-image feature of the face image sample in step 401 is obtained, and the dimension of the full-image feature is w×h×N, where N is the channel dimension. The full-image feature contains the semantic information of the face, so it can be transformed based on the semantic information to obtain a series of corresponding heat maps (i.e., masks) at different facial component positions. The full-image feature is convolutionally transformed to obtain K masks that respond to different facial components, each mask responds at a specific facial component position, and the dimension of each mask is w×h. The set of K masks constitutes the second mask group.

[0126] Step 503: Calculate similarity scores between multiple facial part masks in the second mask group.

[0127] In order to reduce the information overlap between the predicted masks and ensure that they focus on different parts of the face, a regularization term needs to be introduced to constrain the predicted masks to be not too consistent. This constraint is achieved by calculating the similarity score S between the masks. m The calculation method is to first transform each w×h dimensional mask M in the second mask group i Flatten into column vectors, and then combine them in parallel into a w×h×K-dimensional matrix W. Perform singular value decomposition on W to obtain a series of singular values ​​σ1, σ2, …, σ from large to small r .

[0128] Each column vector of the eigenvector matrix obtained by singular value decomposition can be regarded as each pattern component of the mask. The singular value can be regarded as the response intensity of the current mask group on the corresponding pattern component. m The calculation formula is as follows:

[0129]

[0130] Among them, σ1 is the largest singular value, σ j is the jth singular value, and K is the number of masks.

[0131] That is, the negative logarithm of the proportion of the intensity of the most mainstream mode component, S m It is positively correlated with the diversity of masks. When all masks M i When all tend to the same, there is only one mode component, at this time S m =-log K 1=0, when all masks M i When they are completely irrelevant,

[0132] By introducing the similarity between masks, the problem in Problem 1 above that using the cross entropy loss function for training will cause the K masks predicted by the decoupling module to converge, resulting in decoupling failure, is solved.

[0133] Step 504: Based on the target model, predict and generate the posterior distribution of the t-th word.

[0134] Specifically, based on the current dynamic fusion basis state and the t-1th word of the emotion description sentence, a soft attention mechanism is used to fuse the features of the multiple facial parts to obtain the tth high-level semantic feature, and the tth high-level semantic feature is input into the target model to generate the tth word, update the dynamic fusion basis state, and calculate the posterior distribution of the tth word.

[0135] Step 505: Calculate the target distribution of the tth word of the supervised annotation sentence based on the semantic relevance.

[0136] Specifically, based on the semantic relevance between the tth word in the supervised annotation sentence and the words in the candidate vocabulary, the target distribution of the tth word is calculated, where the semantic relevance reflects the similarity information between words.

[0137] By modifying the target distribution that needs to be fitted by the posterior distribution and utilizing the similarity information of synonyms, the convergence problem caused by the small number of training samples for each class in the above problem 2 is solved.

[0138] Step 506: Calculate the distribution difference score of the t-th word according to the posterior distribution of the t-th word and the target distribution of the t-th word.

[0139] Specifically, the KL divergence between the posterior distribution of the t-th word and the target distribution of the t-th word is calculated, and the KL divergence is used as the distribution difference score of the t-th word.

[0140] Step 507: perform an addition operation on t, and start the distribution difference score calculation process for the next word, until the distribution difference scores of all words are obtained.

[0141] Specifically, an addition operation is performed on t, the posterior distribution and target distribution of the next word are calculated, and the distribution difference score of the next word is obtained based on the posterior distribution and the target distribution, and then an addition operation is performed on t until the distribution difference scores of all words are obtained.

[0142] Step 508: Obtain an optimization target according to the distribution difference scores of all words, the similarity scores and the second predefined hyperparameter.

[0143] The second predefined hyperparameter is a parameter that is set manually during training and affects the training strategy. It is not updated during the training process as the optimization process proceeds (i.e., it is not within the range of parameter updates performed by the optimizer based on the loss function). In order to achieve different training effects, researchers or engineers need to adjust the hyperparameters before each training. The second predefined hyperparameter reflects the strength of the restriction on the predicted mask convergence phenomenon.

[0144] Specifically, the distribution difference scores of all words, the similarity scores and the second predefined hyperparameter are substituted into a preset optimization objective function to obtain an optimization target value.

[0145] Step 509: Adjust the parameters in the target model to minimize the value of the optimization target.

[0146] Specifically, the parameters in the target model are adjusted. The adjustment of the parameters in the model will cause a change in the posterior distribution. The change in the posterior distribution will cause a change in the difference score, which in turn affects the value of the optimization target. When the value of the optimization target is minimized, the training is completed.

[0147] Step 510: Save the adjusted parameters to obtain the language generation model.

[0148] Specifically, the model parameter values ​​that minimize the optimization target are saved to obtain the language generation model.

[0149] In the embodiment of the present invention, during the process of training a language generation model, the target distribution that needs to be fitted by the posterior distribution is modified, and the similarity information of synonyms is utilized to solve the convergence problem caused by the small number of training samples for each class. The success rate of decoupling is improved by introducing the similarity between masks, thereby improving the accuracy of the model in determining facial emotions.

[0150] In one embodiment, the step of calculating the target distribution of the tth word of the supervised labeled sentence based on semantic relevance includes the following sub-steps:

[0151] , for the tth word of the supervised annotation sentence, calculate the semantic relevance between the tth word and all words in the pre-stored candidate word list, and normalize the semantic relevance to obtain the similarity distribution corresponding to the tth word;

[0152] Based on the similarity distribution corresponding to the t-th word, the impulse distribution corresponding to the t-th word and a first predefined hyperparameter, a target distribution of the t-th word is calculated.

[0153] Specifically, since there are a large number of synonyms in emotion description terms (such as happy and joy, sad and sad, etc.), if they are treated as independent and mutually exclusive categories, it will cause the same visual semantics to correspond to multiple different words, and there will be fewer samples corresponding to each word, resulting in a slow model convergence speed. Therefore, it is necessary to consider the semantic similarity between different words and use the training samples corresponding to synonyms (converted by a certain coefficient) as the training samples of the current word, so as to enrich the training data and improve the convergence speed.

[0154] In the process of calculating the semantic relevance, we first use the word2vec word vector pre-trained in the massive Chinese corpus as the semantic information representation of each word. We use the inner product operation of the tth word of the supervised annotation sentence and the word vector of all words in the vocabulary to obtain the relevance score, and then perform Softmax normalization on the relevance score to obtain the similarity distribution s t .

[0155] Among them, the first predefined hyperparameter is a parameter that is manually set during training and affects the training strategy. During the training process, it is not updated as the optimization process proceeds (that is, it is not within the range of parameter updates performed by the optimizer based on the loss function). In order to achieve different training effects, researchers or engineers are required to adjust the hyperparameters before each training. The first predefined hyperparameter reflects the strength of the restrictive effect of supervised annotation. The larger the value of the first predefined hyperparameter, the more the synonyms of the current annotated word are likely to be considered as reasonable generation results, and the smaller the value of the first predefined hyperparameter, the more inclined it is to only recognize the annotated words.

[0156] Optionally, the target distribution is calculated as follows:

[0157] P t * =γs t +(1-γ)δ(c t )

[0158] Among them, γ is the first predefined hyperparameter, s t is the similarity distribution of the tth word, δ(c t ) is in c t The impulse distribution at .

[0159] An embodiment of the present invention calculates the semantic relevance between the t-th word and all words in a pre-stored candidate word list, and normalizes the semantic relevance to obtain a similarity distribution corresponding to the t-th word. Based on the similarity distribution corresponding to the t-th word, the impulse distribution corresponding to the t-th word and a first predefined hyperparameter, the target distribution of the t-th word is calculated, and the convergence problem caused by the small number of training samples for each class is solved by optimizing the calculation process of the target distribution.

[0160] In one embodiment, obtaining the optimization target according to the distribution difference scores of all words, the similarity score and the second predefined hyperparameter includes:

[0161] The optimization target is calculated using the optimization target function according to the distribution difference scores of all words, the similarity score and the second predefined hyperparameter.

[0162] Wherein, the optimization objective function is:

[0163] L=∑ t S t +λ×S m

[0164] Among them, S t is the difference score of the tth word, λ is the second predefined hyperparameter, S m Score the familiarity.

[0165] This optimization objective function not only constrains the difference score between the target distribution and the posterior distribution, but also m Constrain the similarity between multiple predicted masks.

[0166] The embodiment of the present invention constrains the difference score between the target distribution and the posterior distribution and the similarity between the masks through a predefined optimization objective function, thereby ensuring the accuracy of the model output statement after training is completed.

[0167] The emotion determination device for a facial image provided by the present invention is described below. The emotion determination device for a facial image described below and the emotion determination method for a facial image described above can be referenced to each other.

[0168] Another embodiment of the present invention provides a device for determining emotion of a facial image, such as Figure 6 As shown, it includes: a face convolution module 610, a face component decoupling module 620, a component relationship reasoning module 630 and a language generation module 640, wherein:

[0169] The face convolution module 610 is used to perform convolution calculation on the face image to be analyzed and extract the full-image features of the face image to be analyzed;

[0170] A facial component decoupling module 620, configured to decouple the full image features to obtain features of multiple facial components;

[0171] A component relationship reasoning module 630, configured to dynamically fuse the features of the plurality of face components based on a dynamic fusion basis state to form a high-level semantic feature;

[0172] The language generation module 640 is used to generate emotion description sentences in a word-by-word prediction manner based on the high-level semantic features and the language generation model.

[0173] Optionally, the component relationship reasoning module 630 is used to initialize the dynamic fusion basis state;

[0174] Initialize the first word of the emotion description sentence to be an empty tag;

[0175] Based on the current dynamic fusion basis state, a soft attention mechanism is used to fuse the features of the multiple face parts to obtain the t-th high-level semantic feature;

[0176] The language generation module 640 is used to input the t-th high-level semantic feature into the language generation model, generate the t-th word, and update the dynamic fusion basis state, perform an addition operation on t, and start generating the next word until all the words included in the emotion description sentence are generated;

[0177] All the words generated by the language generation model are concatenated in order to obtain the emotion description sentence;

[0178] Wherein, t is greater than or equal to 1 and less than or equal to the total number of words contained in the emotion description sentence.

[0179] Optionally, the face component decoupling module 620 is used to:

[0180] Performing a convolution transformation on the full-image feature to obtain a first mask group including a plurality of facial part masks;

[0181] Performing a dimensionality reduction operation on the full-image features to obtain a feature map after dimensionality reduction;

[0182] Expanding the channel dimensions of the plurality of facial part masks in the first mask group in a repeated manner to obtain a plurality of expanded facial part masks, wherein the channel dimensions of the expanded facial part masks are the same as the channel dimensions of the feature map after dimensionality reduction;

[0183] Applying the multiple expanded facial component masks to the feature map after dimensionality reduction to obtain multiple semantic feature maps corresponding to different facial components;

[0184] The multiple semantic feature maps are flattened into vectors, and subjected to feature transformation to obtain features of multiple facial parts of the facial image to be analyzed.

[0185] Optionally, the facial image emotion determination device also includes a model training module, which is used to train a target model based on facial image samples, supervised annotation sentences corresponding to the facial image samples and a predefined optimization objective function, save the parameters of the target model when the training is completed, and obtain the language generation model.

[0186] Optionally, the model training module is specifically used for:

[0187] Obtaining a face image sample and a supervised annotation sentence corresponding to the face image sample;

[0188] Performing convolution calculation on the face image sample to extract full-image features of the face image sample;

[0189] Performing a convolution transformation on the full image features to obtain a second mask group including multiple facial part masks;

[0190] Calculating similarity scores between a plurality of facial part masks in the second mask group;

[0191] Based on the target model, predict the posterior distribution of the tth word;

[0192] Calculate the target distribution of the tth word of the supervised annotation sentence based on the semantic relevance;

[0193] Calculating a distribution difference score of the t-th word according to the posterior distribution of the t-th word and the target distribution of the t-th word;

[0194] Perform an addition operation on t and start the distribution difference score calculation process for the next word until the distribution difference scores of all words are obtained;

[0195] Obtaining an optimization target according to the distribution difference scores of all words, the similarity score and a second predefined hyperparameter;

[0196] Adjusting the parameters in the target model to minimize the value of the optimization target;

[0197] The adjusted parameters are saved to obtain the language generation model.

[0198] Optionally, the calculating the target distribution of the tth word of the supervised labeled sentence based on the semantic relevance includes:

[0199] For the t-th word of the supervised labeled sentence, the semantic relevance between the t-th word and all words in the pre-stored candidate word list is calculated, and the semantic relevance is normalized to obtain a similarity distribution corresponding to the t-th word;

[0200] Based on the similarity distribution corresponding to the t-th word, the impulse distribution corresponding to the t-th word and a first predefined hyperparameter, a target distribution of the t-th word is calculated.

[0201] Optionally, obtaining the optimization target according to the distribution difference scores of all words, the similarity score and the second predefined hyperparameter includes:

[0202] According to the distribution difference scores of all words, the similarity score and the second predefined hyperparameter, an optimization target is calculated using an optimization target function;

[0203] Wherein, the optimization objective function is:

[0204] L=∑ t S t +λ×S m

[0205] Among them, S t is the difference score of the tth word, λ is the second predefined hyperparameter, S m Score the familiarity.

[0206] The facial image emotion determination device provided by the present invention can achieve Figures 1 to 5 The various processes implemented by the method embodiment and achieving the same technical effect are not described here to avoid repetition.

[0207] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the method for determining the emotion of a face image, the method comprising: performing convolution calculation on the face image to be analyzed, and extracting the full-image features of the face image to be analyzed;

[0208] Decoupling the full-image features to obtain features of multiple facial parts;

[0209] Based on the dynamic fusion basis state, dynamically fuse the features of the multiple facial parts to form a high-level semantic feature;

[0210] Based on the high-level semantic features and the language generation model, emotion description sentences are generated in a word-by-word prediction manner.

[0211] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0212] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the method for determining the emotion of a facial image provided by the above methods, the method comprising: performing convolution calculation on the facial image to be analyzed, extracting the full-image features of the facial image to be analyzed;

[0213] Decoupling the full-image features to obtain features of multiple facial parts;

[0214] Based on the dynamic fusion basis state, dynamically fuse the features of the multiple facial parts to form a high-level semantic feature;

[0215] Based on the high-level semantic features and the language generation model, emotion description sentences are generated in a word-by-word prediction manner.

[0216] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned methods for determining emotions of facial images, the methods comprising: performing convolution calculation on the facial image to be analyzed, and extracting full-image features of the facial image to be analyzed;

[0217] Decoupling the full-image features to obtain features of multiple facial parts;

[0218] Based on the dynamic fusion basis state, dynamically fuse the features of the multiple facial parts to form a high-level semantic feature;

[0219] Based on the high-level semantic features and the language generation model, emotion description sentences are generated in a word-by-word prediction manner.

[0220] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0221] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eos>

Claims

1. A method for determining emotion of a facial image, characterized in that: include: Perform convolution calculation on the face image to be analyzed to extract full-image features of the face image to be analyzed; Decoupling the full-image features to obtain features of multiple facial parts; Based on the dynamic fusion basis state, dynamically fuse the features of the multiple facial parts to form a high-level semantic feature; Based on the high-level semantic features and the language generation model, generate emotion description sentences in a word-by-word prediction manner; The method of dynamically fusing the features of the plurality of face components based on the dynamic fusion state to form a high-level semantic feature includes: Initialize dynamic fusion based on state; Initialize the first word of the emotion description sentence to be an empty tag; Based on the current dynamic fusion basis state, a soft attention mechanism is used to fuse the features of the multiple face parts to obtain the first t high-level semantic features; The generating of emotion description sentences in a word-by-word prediction manner based on the high-level semantic features and the language generation model includes: The said t The high-level semantic features are input into the language generation model to generate the t words, and update the dynamic fusion based on the state. t Execute the plus 1 operation to start the generation of the next word until all the words included in the emotion description sentence are generated; the dynamic fusion basis state is updated according to the sequence information of the generated words, the full-image features and the current dynamic fusion basis state; All words generated by the language generation model are concatenated in order to obtain the emotion description sentence; in, t is greater than or equal to 1 and less than or equal to the total number of words contained in the emotion description sentence; Decoupling the full image features to obtain features of multiple face parts includes: Performing a convolution transformation on the full image feature to obtain a first mask group including a plurality of facial component masks; the masks are corresponding heat maps obtained at different facial component positions; Performing a dimensionality reduction operation on the full-image features to obtain a feature map after dimensionality reduction; Expanding the channel dimensions of the plurality of facial part masks in the first mask group in a repeated manner to obtain a plurality of expanded facial part masks, wherein the channel dimensions of the expanded facial part masks are the same as the channel dimensions of the feature map after dimensionality reduction; Applying the multiple expanded facial component masks to the feature map after dimensionality reduction to obtain multiple semantic feature maps corresponding to different facial components; The multiple semantic feature maps are flattened into vectors, and subjected to feature transformation to obtain features of multiple facial parts of the facial image to be analyzed.

2. The method for determining emotion of a facial image according to claim 1, characterized in that: Also includes: The target model is trained according to the face image samples, the supervised annotation sentences corresponding to the face image samples and the predefined optimization objective function, and the parameters of the target model when the training is completed are saved to obtain the language generation model.

3. The method for determining emotion of a facial image according to claim 2, characterized in that: The target model is trained according to the training set images, the supervised annotation sentences corresponding to the training set images, and the predefined optimization objective function, and the parameters of the target model are saved when the training is completed, and the language generation model is obtained, including: Obtaining a face image sample and a supervised annotation sentence corresponding to the face image sample; Performing convolution calculation on the face image sample to extract full-image features of the face image sample; Performing a convolution transformation on the full image features to obtain a second mask group including multiple facial part masks; Calculating similarity scores between a plurality of facial part masks in the second mask group; Based on the target model, the prediction generates the t The posterior distribution of words; The first sentence of the supervised annotation is calculated based on the semantic relevance t Target distribution of words; According to the said t The posterior distribution of the word and the t The target distribution of words is calculated t The distribution difference score of each word; right t Perform a plus-one operation to start the distribution difference score calculation process for the next word until the distribution difference scores of all words are obtained; Obtaining an optimization target according to the distribution difference scores of all words, the similarity score and a second predefined hyperparameter; Adjusting the parameters in the target model to minimize the value of the optimization target; The adjusted parameters are saved to obtain the language generation model.

4. The method for determining emotion of a facial image according to claim 3, characterized in that: The first step of calculating the supervised annotation sentence based on the semantic relevance t The target distribution of words includes: For the supervised labeled sentence t words, calculate the t The semantic relevance of the word with all the words in the pre-stored candidate word list is normalized to obtain the first t Similarity distribution corresponding to each word; Based on the t The similarity distribution corresponding to the words, t The impulse distribution corresponding to the word and the first predefined hyperparameter are used to calculate the first t The target distribution of words.

5. The method for determining emotion of a facial image according to claim 3, characterized in that: The optimization target is obtained according to the distribution difference scores of all words, the similarity score and the second predefined hyperparameter, including: According to the distribution difference scores of all words, the similarity score and the second predefined hyperparameter, an optimization target is calculated using an optimization target function; Wherein, the optimization objective function is: ； in, S t For the t The difference score of the words, is the second predefined hyperparameter, is the similarity score.

6. A device for determining emotion of a facial image, characterized in that: include: A face convolution module, used to perform convolution calculation on the face image to be analyzed and extract the full-image features of the face image to be analyzed; A facial component decoupling module, used for decoupling the full image features to obtain features of multiple facial components; A component relationship reasoning module, used for dynamically fusing the features of the plurality of face components based on a dynamic fusion basis state to form a high-level semantic feature; A language generation module, used to generate emotion description sentences in a word-by-word prediction manner based on the high-level semantic features and the language generation model; The method of dynamically fusing the features of the plurality of face components based on the dynamic fusion state to form a high-level semantic feature includes: Initialize dynamic fusion based on state; Initialize the first word of the emotion description sentence to be an empty tag; Based on the current dynamic fusion basis state, a soft attention mechanism is used to fuse the features of the multiple face parts to obtain the first t high-level semantic features; The generating of emotion description sentences in a word-by-word prediction manner based on the high-level semantic features and the language generation model includes: The said t The high-level semantic features are input into the language generation model to generate the t words, and update the dynamic fusion based on the state. t Execute the plus 1 operation to start the generation of the next word until all the words included in the emotion description sentence are generated; the dynamic fusion basis state is updated according to the sequence information of the generated words, the full-image features and the current dynamic fusion basis state; All words generated by the language generation model are concatenated in order to obtain the emotion description sentence; in, t is greater than or equal to 1 and less than or equal to the total number of words contained in the emotion description sentence; Decoupling the full image features to obtain features of multiple face parts includes: Performing a convolution transformation on the full image feature to obtain a first mask group including a plurality of facial component masks; the masks are corresponding heat maps obtained at different facial component positions; Performing a dimensionality reduction operation on the full-image features to obtain a feature map after dimensionality reduction; Expanding the channel dimensions of the plurality of facial part masks in the first mask group in a repeated manner to obtain a plurality of expanded facial part masks, wherein the channel dimensions of the expanded facial part masks are the same as the channel dimensions of the feature map after dimensionality reduction; Applying the multiple expanded facial component masks to the feature map after dimensionality reduction to obtain multiple semantic feature maps corresponding to different facial components; The multiple semantic feature maps are flattened into vectors, and subjected to feature transformation to obtain features of multiple facial parts of the facial image to be analyzed.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for determining the emotion of a facial image as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for determining the emotion of a facial image as claimed in any one of claims 1 to 5 are implemented.

Images