Image-based emotion recognition method, device, equipment and storage medium

Abstract

The application provides an emotion recognition method and device based on images, equipment and a storage medium, relates to the technical field of deep learning, and comprises the following steps: obtaining a target image and a description text of the target image by a target user, the target image being an image that can cause the user to produce an emotion; inputting the target image into an image caption generation model to generate an image caption, so as to obtain an image caption of multiple emotional styles corresponding to the target image; determining a first PAD value corresponding to the image caption of each emotional style in the image captions of multiple emotional styles; determining a second PAD value corresponding to the description text; determining a target PAD value according to the multiple first PAD values and the second PAD value, and then determining an emotional state of the target user according to the target PAD value, so as to improve the accuracy of the recognition result. In addition, the obtained image captions of multiple emotional styles can further improve the accuracy of the recognition result of the emotional state of the user.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to an image-based emotion recognition method, apparatus, device, and storage medium. Background Technology

[0002] With the development of deep learning, it has not only greatly promoted the development of artificial intelligence and machine learning, but also promoted the development of other fields, giving rise to interdisciplinary research areas between deep learning and other fields, such as the technical research area at the intersection of deep learning and psychology. Among these, the study of emotional data in psychology is a major research direction, mainly studying human emotional states, such as positive and negative emotions.

[0003] In related technologies, research on emotional states typically begins with users self-scoring on a standardized scale, followed by professionals assessing the user's emotional state based on their scores. This method is heavily influenced by the subjective thinking of both users and professionals, making it time-consuming, labor-intensive, and prone to low accuracy. Summary of the Invention

[0004] This application provides an image-based emotion recognition method, apparatus, device, and storage medium to address the problem of low accuracy in existing emotion recognition methods.

[0005] In a first aspect, this application provides an image-based emotion recognition method, comprising:

[0006] Obtain the target image and the target user's descriptive text for the target image. The target image is one that can evoke emotions in the user.

[0007] The target image is input into the image caption generation model to generate image captions, resulting in image captions with various emotional styles corresponding to the target image.

[0008] For each emotional style of image caption in multiple emotional style image captions, determine the first PAD value corresponding to the emotional style image caption; and determine the second PAD value corresponding to the descriptive text.

[0009] The target PAD value is determined based on multiple first PAD values ​​and second PAD values.

[0010] Determine the target user's emotional state based on the target PAD value.

[0011] In one possible implementation, determining a target PAD value based on multiple first PAD values ​​and second PAD values ​​includes: weighting the multiple first PAD values ​​and second PAD values ​​according to a preset formula to obtain the target PAD value; wherein the preset formula is:

[0012]

[0013] Among them, y i a represents the first PAD value corresponding to the image caption of the i-th mood style. i R0 represents the scale score corresponding to the i-th emotional style, N represents the total number of emotional styles, and A represents the normalization coefficient.

[0014] In one possible implementation, determining the first PAD value corresponding to the image caption of the emotional style includes: determining a first semantic vector corresponding to the image caption of the emotional style, the first semantic vector containing multiple first semantic elements; for each of the multiple first semantic elements, determining a first semantic similarity between the first semantic element and each second semantic element in the second semantic vector, the second semantic vector being determined based on an emotion dictionary, the emotion dictionary including multiple emotion phrases; among the second semantic elements, determining a first target semantic element whose first semantic similarity to the first semantic element satisfies a first set condition; and weighting the first semantic similarity corresponding to the first target semantic element and the PAD value corresponding to the first target semantic element to obtain the first PAD value.

[0015] In one possible implementation, determining the second PAD value corresponding to the descriptive text includes: determining a third semantic vector corresponding to the descriptive text, the third semantic vector containing multiple third semantic elements; for each of the multiple third semantic elements, determining a second semantic similarity between the third semantic element and each second semantic element in the second semantic vector; among the second semantic elements, determining a second target semantic element whose second semantic similarity to the third semantic element satisfies a second predetermined condition; and weighting the second semantic similarity corresponding to the second target semantic element and the PAD value corresponding to the second target semantic element to obtain the second PAD value.

[0016] In one possible implementation, the image captioning generation model in the image-based emotion recognition method may include a convolutional neural network and an attention-based stylized text generation module. The stylized text generation module includes a GRU unit, an encoding submodule, and a decoding submodule. The target image is input into the image captioning generation model to generate image captions corresponding to various emotion styles, including: inputting the target image into the convolutional neural network for feature extraction to obtain image features corresponding to the target image; inputting the image features into the GRU unit for semantic extraction to obtain a fourth semantic vector; inputting the fourth semantic vector into the encoding submodule for encoding processing to obtain an encoded vector; inputting the encoded vector into the decoding submodule for decoding processing to obtain a decoded vector; and concatenating the encoded vector and the decoded vector processed by the attention mechanism to obtain image captions corresponding to various emotion styles of the target image.

[0017] In one possible implementation, the image captioning generation model in the image-based emotion recognition method can be trained as follows: Sample data is acquired, including sample images and corresponding reference image captions, as well as stylized text data, including stylized text and corresponding reference style tags; the sample images are input into the image captioning generation model to obtain output image captions corresponding to the sample images; based on style indicators and the stylized text data, the output image captions are adjusted; a first loss value is determined between the adjusted output image captions and the reference image captions; and the image captioning generation model is adjusted based on the first loss value.

[0018] In one possible implementation, the image-based emotion recognition method further includes: using a classification model to perform stylized text classification on generated image captions with multiple emotion styles, to determine whether the image captions conform to the style of the target image. The classification model can be trained as follows: acquiring stylized text data; inputting the stylized text from the stylized text data into the classification model to obtain the output style label corresponding to the stylized text; determining a second loss value between the output style label and a reference style label in the stylized text data; and adjusting the classification model based on the second loss value.

[0019] Secondly, this application provides an image-based emotion recognition device, comprising:

[0020] The acquisition module is used to acquire the target image and the target user's descriptive text for the target image. The target image is an image that can evoke emotions in the user.

[0021] The generation module is used to input the target image into the image caption generation model to generate image captions, resulting in image captions with various emotional styles corresponding to the target image.

[0022] The first determining module is used to determine the first PAD value corresponding to each emotional style of the image caption in multiple emotional style image captions; and to determine the second PAD value corresponding to the descriptive text.

[0023] The second determining module is used to determine the target PAD value based on multiple first PAD values ​​and second PAD values;

[0024] The third determination module is used to determine the emotional state of the target user based on the target PAD value.

[0025] In one possible implementation, the second determining module may be specifically used to: weight multiple first PAD values ​​and second PAD values ​​based on a preset formula to obtain a target PAD value. The preset formula is:

[0026]

[0027] Among them, y i a represents the first PAD value corresponding to the image caption of the i-th mood style. i R0 represents the scale score corresponding to the i-th emotional style, N represents the total number of emotional styles, and A represents the normalization coefficient.

[0028] In one possible implementation, the first determining module may be specifically used to: determine a first semantic vector corresponding to an image caption with an emotional style, the first semantic vector containing multiple first semantic elements; for each of the multiple first semantic elements, determine a first semantic similarity between the first semantic element and each second semantic element in the second semantic vector, the second semantic vector being determined based on an emotion dictionary, the emotion dictionary including multiple emotion phrases; among the second semantic elements, determine a first target semantic element whose first semantic similarity to the first semantic element satisfies a first set condition; and perform weighted processing on the first semantic similarity corresponding to the first target semantic element and the PAD value corresponding to the first target semantic element to obtain a first PAD value.

[0029] In one possible implementation, the first determining module may further be used to: determine a third semantic vector corresponding to the descriptive text, the third semantic vector containing multiple third semantic elements; for each of the multiple third semantic elements, determine a second semantic similarity between the third semantic element and each second semantic element in the second semantic vector; among the second semantic elements, determine a second target semantic element whose second semantic similarity with the third semantic element satisfies a second set condition; and perform weighted processing on the second semantic similarity corresponding to the second target semantic element and the PAD value corresponding to the second target semantic element to obtain a second PAD value.

[0030] In one possible implementation, the image caption generation model in the image-based emotion recognition device may include a convolutional neural network and an attention-based stylized text generation module. The stylized text generation module includes a GRU unit, an encoding submodule, and a decoding submodule. Specifically, the generation module in the image-based emotion recognition device may be used to: input the target image into the convolutional neural network for feature extraction to obtain image features corresponding to the target image; input the image features into the GRU unit for semantic extraction to obtain a fourth semantic vector; input the fourth semantic vector into the encoding submodule for encoding processing to obtain an encoded vector; input the encoded vector into the decoding submodule for decoding processing to obtain a decoded vector; and concatenate the encoded vector and the decoded vector processed by the attention mechanism to obtain image captions with multiple emotion styles corresponding to the target image.

[0031] In one possible implementation, the image captioning generation model in the image-based emotion recognition device can be trained as follows: acquiring sample data, including sample images and corresponding reference image captions, and stylized text data, including stylized text and corresponding reference style tags; inputting the sample images into the image captioning generation model to obtain output image captions corresponding to the sample images; adjusting the output image captions based on style indicators and stylized text data; determining a first loss value between the adjusted output image captions and the reference image captions; and adjusting the image captioning generation model based on the first loss value.

[0032] In one possible implementation, the image-based emotion recognition device further includes: using a classification model to perform stylized text classification on generated image captions with multiple emotion styles, to determine whether the image captions conform to the style of the target image. The classification model can be trained by: acquiring stylized text data; inputting the stylized text from the stylized text data into the classification model to obtain the output style label corresponding to the stylized text; determining a second loss value between the output style label and a reference style label from the stylized text data; and adjusting the classification model based on the second loss value.

[0033] Thirdly, this application provides an electronic device, including a memory and a processor. The memory is used to store program instructions; the processor is used to invoke the program instructions in the memory to execute the image-based emotion recognition method of the first aspect.

[0034] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the image-based emotion recognition method of the first aspect.

[0035] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, is used to implement the image-based emotion recognition method of the first aspect.

[0036] The image-based emotion recognition method, apparatus, device, and storage medium provided in this application acquire a target image and descriptive text from a target user for the target image, wherein the target image is an image that can evoke emotions in the user; the target image is input into an image captioning generation model to generate image captions, resulting in image captions with multiple emotion styles corresponding to the target image; for each emotion style of the image captions, a first PAD value corresponding to the emotion style of the image caption is determined; and a second PAD value corresponding to the descriptive text is determined; based on multiple first PAD values ​​and second PAD values, a target PAD value is determined; and based on the target PAD value, the emotional state of the target user is determined. Specifically, by using an image captioning generation model, image captions with multiple emotion styles are obtained while maintaining the accuracy of semantic vectors, solving the problem of single image caption stylization and making it more conducive to recognizing different emotional states of different users; furthermore, determining the target PAD value based on multiple first PAD values ​​and second PAD values, i.e., combining image modality and text modality to determine the target PAD value, and recognizing the user's emotional state based on the target PAD value, helps improve the accuracy of the recognition results. Attached Figure Description

[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0038] Figure 1 This is a schematic diagram illustrating an application scenario provided in one embodiment of this application;

[0039] Figure 2 A schematic flowchart illustrating an image-based emotion recognition method provided in an embodiment of this application;

[0040] Figure 3 A schematic diagram illustrating the image caption generation process provided in an embodiment of this application;

[0041] Figure 4 A flowchart illustrating an image-based emotion recognition method provided in another embodiment of this application;

[0042] Figure 5 A schematic diagram illustrating model training for an image-based emotion recognition method provided in an embodiment of this application;

[0043] Figure 6 A schematic diagram of the structure of an image-based emotion recognition device provided in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0045] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] The terms “first,” “second,” etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, products, or apparatus.

[0048] With the rapid development of deep learning, some researchers have begun to apply it to the field of psychology. However, research on emotion state recognition in psychology still mainly relies on standard scale scoring, lacking image-based methods for emotion state recognition or judgment.

[0049] To address the aforementioned issues, this application proposes an image-based emotion recognition method. This method introduces a stylized image captioning generation model, which, while ensuring the semantic accuracy of image captions, enables the generation of image captions in different styles, thus solving the problem of a single stylistic approach for image captions. Furthermore, by combining the generated stylized image captions with the user's textual description of the image, and based on the PAD 3D emotion model, it achieves user emotion state recognition. This innovative approach combines image and text modalities to identify user emotion states, thereby improving the accuracy of the recognition results.

[0050] Image captioning is a major task in deep learning today, typically involving multiple aspects such as computer vision and natural language processing. Simply put, image captioning is about summarizing an image into text; that is, describing the targets in the image, the information contained within those targets, and the relationships between them using text. Specifically, this task requires detecting targets in an image, obtaining the target information contained within those targets, understanding the relationships between targets based on that information, and then expressing the image using appropriate language based on the obtained target information and the relationships between targets.

[0051] Current image captioning generation tasks have two main improvement goals: improving the accuracy of semantic representation and stylizing image captions. While significant progress has been made in improving the accuracy of semantic representation, most work on stylized image captioning focuses on single-style stylized captions, lacking resources for generating captions with multiple stylization types. Therefore, how to generate stylized image captions more efficiently is one of the key focuses of this application.

[0052] Furthermore, image captioning generation methods can be introduced into other disciplines, such as psychology, becoming an important research area at the intersection of deep learning and psychology. Specifically, the generated stylized image captions can be applied to psychology for emotion state recognition. For instance, using transfer learning, emotional images can be converted into text information, and then text sentiment analysis can be performed to identify the user's emotional state when viewing the emotional image. Here, emotional images refer to images that can evoke emotions in users.

[0053] In addition, when combining generated stylized image captions for emotion state recognition, the research focus of this application is on the construction of the overall emotion state recognition process and the design of the neural network framework.

[0054] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, this application scenario includes image 11, user 12, and computer 13. When user 12 sees image 11, they generally experience an emotional change, and descriptive text about image 11 is generated. Image 11 and descriptive text can be input into computer 13 together. The memory in computer 13 can store image 11, descriptive text, and program instructions for processing image 11 and descriptive text accordingly. Accordingly, the processor in computer 13 can call the relevant program instructions to process the image 11 and descriptive text stored in memory and obtain the emotional state recognition result for user 12.

[0055] For example, the number of images 11 and users 12 can both be at least one. In practical applications, when multiple users simultaneously express emotional changes in response to image 11 and provide textual descriptions, the processor in computer 13 can invoke relevant program instructions to perform batch processing, quickly identify the emotional states of multiple users, and improve recognition efficiency.

[0056] It should be noted that computer 13 can also be replaced by a laptop, server cluster, or other computing devices with a certain computing power. Alternatively, users can take photos with their mobile phones, describe the content of the photos with text, and upload them to the cloud, where the cloud server can call relevant program instructions to perform emotion state recognition, etc.

[0057] The following is combined Figure 1 Application scenarios, refer to Figure 2 This application describes an image-based emotion recognition method according to exemplary embodiments thereof. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not intended to be construed as... Figure 1 The limitations of the application scenarios shown.

[0058] Figure 2 This is a schematic flowchart illustrating an image-based emotion recognition method provided in an embodiment of this application. Figure 2 As shown, the image-based emotion recognition method in this application embodiment includes the following steps:

[0059] S201: Obtain the target image and the target user's descriptive text for the target image. The target image is an image that can evoke emotions in the user.

[0060] In this step, the target image can be obtained through devices with photographic capabilities, such as mobile phones or cameras, or downloaded directly from the internet. Additionally, target users can be identified, and descriptive text can be generated from the obtained target image.

[0061] After obtaining the target image and the target user's descriptive text about the target image, subsequent actions such as identifying the target user's emotional state can be performed.

[0062] For example, a target user might describe a target image, and after obtaining a descriptive text, the target user's emotional state could be identified; multiple target users might describe a target image, and after obtaining multiple descriptive texts, the emotional states of multiple target users could be identified simultaneously; or the same target user might describe multiple target images, and after obtaining multiple descriptive texts, the emotional state of that target user could be identified, and so on.

[0063] It is understandable that, in reality, different target users will experience different emotional states when viewing the same target image. For example, after viewing multiple positive-style target images, the same target user may shift from a negative emotional state to a positive one; conversely, after viewing multiple negative-style target images, the same target user may shift from a positive emotional state to a negative one.

[0064] Alternatively, a target image can be composed of multiple images of different styles. These images are then viewed by different target users, and the changes in their emotional state from the first image to the last can be analyzed to determine the stability of their emotional state. The different image styles include positive, neutral, and negative images, among others.

[0065] For example, the target image can also be replaced with a video or other data or materials that evoke emotions in the user.

[0066] S202: Input the target image into the image caption generation model to generate image captions, and obtain image captions with various emotional styles corresponding to the target image.

[0067] For example, image captions with various emotional styles can include positive, neutral, and negative image captions. Further subdivided, positive styles can include joy, optimism, ease, and surprise; neutral styles can include mildness and boredom; and negative styles can include sadness, fear, anxiety, contempt, disgust, and resentment.

[0068] In practice, emotional styles are more diverse. We can classify emotional styles into coarse-grained and fine-grained categories based on the actual situation and specific application scenarios. There are no restrictions here.

[0069] S203: For each type of image caption in multiple mood styles, determine the first PAD value corresponding to the mood style image caption; and determine the second PAD value corresponding to the descriptive text.

[0070] Specifically, the PAD value is a three-dimensional vector that can be obtained through mapping using the PAD three-dimensional emotion model. The PAD three-dimensional emotion model is a commonly used model that uses dimensional space to identify emotional states, and it has wide applications in audio / video speech synthesis, emotion computing, and other fields. The PAD three-dimensional emotion model includes three dimensions: Pleasure, Arousal, and Dominance.

[0071] Among them, pleasure can reflect the positive (positive) and negative (negative) aspects of the user's emotional (or mood) state; activation can reflect the activation level and excitement level of the user's nervous (or physiological) system, with high excitement being positive and low excitement being negative; dominance can reflect the strength of the mutual dominance between the user and the external environment, with user dominance being positive and external dominance being negative.

[0072] In conjunction with the above embodiments, for each emotional style of image caption in multiple emotional style image captions, determining the first PAD value corresponding to the emotional style image caption may include: first determining the semantic vector (or semantic sequence, semantic primitive sequence) corresponding to the emotional style image caption; then mapping the semantic vector to the PAD three-dimensional emotional space, thereby obtaining the first PAD value corresponding to the emotional style image caption.

[0073] For example, for positive-style image captions, the semantic vector of the image caption is extracted, and then the semantic vector is mapped to the PAD three-dimensional sentiment space through the PAD three-dimensional sentiment model to obtain the PAD value corresponding to the positive-style image caption. Similarly, the PAD value corresponding to neutral-style image captions, negative-style image captions, and descriptive text can be determined.

[0074] S204: Determine the target PAD value based on multiple first PAD values ​​and second PAD values.

[0075] In some examples, determining the target PAD value based on multiple first PAD values ​​and second PAD values ​​may include: weighting the multiple first PAD values ​​and second PAD values ​​according to a preset formula to obtain the target PAD value; wherein the preset formula is:

[0076]

[0077] Among them, y i a represents the first PAD value corresponding to the image caption of the i-th mood style. i Ri represents the scale score corresponding to the i-th emotional style, R0 represents the second PAD value, N represents the total number of emotional style categories, and A represents the normalization coefficient. N can be determined according to the actual situation; for example, when emotional styles include positive, neutral, and negative styles, N = 3. The scale score is the user's rating of the generated image captions. If the rating mechanism is a 5-point scale, then the normalization coefficient A is N × 5 = 3 × 5 = 15.

[0078] R IThis represents the target PAD value of the target image I obtained based on a preset formula. The target PAD value is obtained by weighting the first PAD value corresponding to the image caption and the second PAD value corresponding to the descriptive text, which better reflects the actual situation and the emotional state of the target user, effectively improving the accuracy of emotion state recognition results.

[0079] When the scale's scoring mechanism is a 5-point system, the specific scoring principles can be as follows: 5 points are awarded for language fluency, accurate word choice, and relevance to the actual scenario and the user's inner thoughts; 4 points are awarded for language fluency, relevance to the actual scenario and the user's inner thoughts, but inaccurate word choice; 3 points are awarded for language fluency and relevance to the actual scenario, but inaccurate word choice and inability to accurately describe the user's inner thoughts; 2 points are awarded for language fluency alone; 1 point is awarded for language incoherence or ambiguity; and so on.

[0080] In some examples, compared with methods for identifying emotional states based on traditional standard scales, the method provided in this application, which is based on a three-dimensional PAD emotion model and determines a target PAD value based on multiple first PAD values ​​and second PAD values, i.e., combining image modality and text modality to determine the target PAD value and thus identify the user's emotional state, still has good validity (or effectiveness).

[0081] S205: Determine the target user's emotional state based on the target PAD value.

[0082] Specifically, in the PAD three-dimensional sentiment model, the coordinates of PAD parameters can be determined using a set of standard sentiment scales. Each sentiment (or emotional state) uniquely corresponds to a PAD three-dimensional sentiment space coordinate position. After the PAD parameters are normalized, the emotional state can be identified by a unique three-dimensional coordinate, providing a high-confidence evaluation. Furthermore, the independence between the various dimensions of PAD makes it easier to distinguish text sentiments or user emotional states located in different sentiment dimensions.

[0083] For example, since each emotional state uniquely corresponds to a PAD three-dimensional emotional space coordinate position, that is, there is a one-to-one correspondence between the target PAD value and the target user's emotional state, the target user's emotional state can be determined after the target PAD value is determined.

[0084] In conjunction with the above embodiments, determining the target user's emotional state based on the target PAD value can include: based on the preset correspondence between pleasure value (P value), activation value (A value), and dominance value (D value) and PAD values, determining a reference PAD value corresponding to the target PAD value based on the pleasure value, activation value, and dominance value of the target PAD value; and determining the target user's emotional state based on the reference PAD value. The preset correspondence between pleasure value, activation value, and dominance value and PAD value can be determined by setting up an emotional state recognition experiment. That is, collecting a large amount of image data of different styles, conducting a questionnaire survey based on the PAD standard scale, obtaining different users' scores on the PAD standard scale, taking the average, and obtaining the reference PAD value and the corresponding emotional state based on the PAD standard scale. Further, the obtained target PAD value can be compared with the reference PAD value to complete the identification of the emotional state.

[0085] Table 1 shows the correspondence between some basic emotional states and PAD values ​​in some examples:

[0086] Table 1

[0087] Serial Number Emotional state p-value A value D value 1 joy 2.77 1.21 1.42 2 optimism 2.48 1.05 1.75 3 easy 2.19 -0.66 1.05 4 Surprise 1.72 1.71 0.22 5 mild 1.57 -0.79 0.38 6 rely 0.39 -0.81 -1.48 7 boring -0.53 -1.25 -0.84 8 sad -0.89 0.17 -0.70 9 fear -0.93 1.30 -0.64 10 anxiety -0.95 0.32 -0.63 11 contempt -1.58 0.32 1.02 12 disgust -1.80 0.40 0.67 13 resentment -1.98 1.10 0.60 14 hostility -2.08 1.00 1.12

[0088] When the target PAD value is (1.72, 1.71, 0.22), referring to Table 1, the target user's emotional state can be determined to be surprise; when the target PAD value is (-0.53, -1.25, -0.84), referring to Table 1, the target user's emotional state can be determined to be bored. Alternatively, when the target PAD value is (1.55, -0.80, 0.38), referring to Table 1, there is no value exactly matching the target PAD value. However, through some similarity algorithms or direct comparison, it can be seen that (1.57, -0.79, 0.38) is closest to the target PAD value (1.55, -0.80, 0.38), therefore, the target user's emotional state can be determined to be mild; and so on.

[0089] The image-based emotion recognition method provided in this application, by using an image captioning generation model, generates image captions with multiple emotion styles while maintaining the accuracy of semantic vectors. This solves the problem of monotonous image captioning styles and is more conducive to recognizing different emotional states of different users. Furthermore, by determining a target PAD value based on multiple first PAD values ​​and second PAD values—that is, by combining image and text modalities—and recognizing the user's emotional state based on the target PAD value, the accuracy of the recognition results is improved.

[0090] In some examples, determining the first PAD value corresponding to the image caption of the emotional style may include: determining a first semantic vector corresponding to the image caption of the emotional style, the first semantic vector containing multiple first semantic elements; for each of the multiple first semantic elements, determining the first semantic similarity between the first semantic element and each second semantic element in the second semantic vector, the second semantic vector being determined based on an emotion dictum, which includes multiple emotion phrases; among the second semantic elements, determining a first target semantic element whose first semantic similarity to the first semantic element satisfies a first set condition; and weighting the first semantic similarity corresponding to the first target semantic element and the PAD value corresponding to the first target semantic element to obtain the first PAD value. The emotion dictum can collect emotion phrases related to emotions based on actual situations, such as joy, pleasure, happiness, excitement, anger, boredom, depression, irritability, and anxiety, etc. Correspondingly, these emotion phrases all have corresponding PAD values.

[0091] Optionally, the first semantic vector corresponding to the mood-style image caption can be determined using Natural Language Processing (NLP) methods, specifically through a Recurrent Neural Network (RNN). Alternatively, it can be determined using models such as word2vec, Neural Network Language Model (NNLM), or Long Short-Term Memory (LSTM).

[0092] For example, to comprehensively consider the influence of multiple emotional states on PAD values, semantic similarity can be introduced. The semantic similarity between multiple first semantic elements in the first semantic vector and multiple second semantic elements in the emotion dictionary is calculated, resulting in multiple semantic similarities. The emotional state with the highest semantic similarity is taken as the primary emotion, and then the PAD values ​​of image captions with different emotional styles are calculated through weighted summation. Semantic similarity can be obtained by calculating the Euclidean distance between the first and second semantic elements, or by calculating the cosine value between them; this is not limited here. For instance, when determining semantic similarity by calculating the cosine value between the first and second semantic elements, the range of the cosine value is [0,1], and the larger the cosine value, the greater the semantic similarity.

[0093] In one example, if the input target image includes elements such as blue sky, white clouds, green mountains, and clear water, the image captioning model generates the following captions: "Blue sky and white clouds symbolize freedom and joy; green mountains and clear water symbolize vitality and hope." By extracting semantic vectors and calculating the semantic similarity between multiple first semantic elements in the image captions and multiple second semantic elements in the emotion dictionary, the model determines that "joy" has the highest semantic similarity to "happiness" in the emotion dictionary, and therefore "happiness" is chosen as the primary emotion.

[0094] In addition, to comprehensively consider the impact of various emotional states on PAD values, when multiple semantic similarities are obtained, conditions can be set to select from these multiple semantic similarities. For example, the first preset condition is to select the top five with the highest semantic similarity.

[0095] After determining semantic similarity, the second semantic element in the emotion dictionary corresponding to the semantic similarity can be identified as the target semantic element. For example, "joy" in the emotion dictionary is the target semantic element. If five semantic similarity values ​​are determined, then there are five target semantic elements. Accordingly, based on the correspondence between emotional states and PAD values, the PAD values ​​corresponding to the five target semantic elements can be obtained respectively. Then, these five semantic similarity values ​​and the five PAD values ​​are multiplied and added together, i.e., weighted, to obtain the first PAD value.

[0096] Based on the above embodiments, the method provided in this application can generate image captions of various emotional styles based on a pre-built and trained image caption generation model and the input image, and calculate the mapping value of the generated image captions in the PAD three-dimensional emotional space, i.e., the PAD value, based on semantic similarity.

[0097] Corresponding to the above embodiments, determining the second PAD value corresponding to the descriptive text may include: determining a third semantic vector corresponding to the descriptive text, wherein the third semantic vector contains multiple third semantic elements; for each of the multiple third semantic elements, determining the second semantic similarity between the third semantic element and each second semantic element in the second semantic vector; among the second semantic elements, determining a second target semantic element whose second semantic similarity with the third semantic element satisfies a second set condition; and weighting the second semantic similarity corresponding to the second target semantic element and the PAD value corresponding to the second target semantic element to obtain the second PAD value. Here, the descriptive text is the text described by the target user for the target image. For example, if the target image includes blue sky, white clouds, green mountains, and green water, the text described by the target user for the target image might be: "When I see this image, I feel a sense of freedom and unconscious happiness."

[0098] In addition, the second setting condition can also be to select the top five with the highest semantic similarity; or it can be set according to the actual situation, without restriction here.

[0099] Furthermore, by extracting semantic vectors, the semantic similarity between multiple third semantic elements in the descriptive text and multiple second semantic elements in the emotion dictionary is calculated. The semantic similarity between "happy" and "optimistic" in the emotion dictionary is found to be the highest, thus "optimistic" is selected as the primary emotion. Similarly, the subsequent calculation of the second PAD value is similar to the calculation of the first PAD value in the above embodiment, and will not be repeated here.

[0100] In some examples, the image captioning generation model in image-based emotion recognition methods may include a convolutional neural network and an attention-based stylized text generation module. The stylized text generation module comprises a GRU unit, an encoding submodule, and a decoding submodule. The target image is input into the image captioning generation model to generate image captions corresponding to various emotion styles. This process includes: inputting the target image into a convolutional neural network for feature extraction to obtain image features; inputting the image features into a GRU unit for semantic extraction to obtain a fourth semantic vector; inputting the fourth semantic vector into the encoding submodule for encoding to obtain an encoded vector; inputting the encoded vector into the decoding submodule for decoding to obtain a decoded vector; and concatenating the encoded vector and the decoded vector processed by the attention mechanism to obtain image captions corresponding to various emotion styles of the target image.

[0101] Figure 3 This is a schematic diagram illustrating an image caption generation process according to an embodiment of this application. Figure 3 As shown, the input target image is processed by an image feature extraction module to extract its features. This module can be a standard convolutional neural network or Inception V3, which offers superior image feature extraction capabilities. After obtaining the image features, a gated recurrent unit (GRU) is used to transform them into a semantic vector {X1, X2, ..., X...}. M Among them, the GRU unit has a better ability to process time sequences. Figure 3 middle <sos>It is a marker indicating the start of a semantic sequence.

[0102] The resulting semantic vector {X1, X2, ..., X} M The input encoding submodule performs encoding processing to obtain the encoded vector {h}. enc,0 h enc,1 , ..., h enc,M }; then encode the vector {h enc,0 h enc,1 , ..., h enc,M The input decoding submodule performs decoding processing to obtain the decoding vector {h}. dec,0 h dec,1 , ..., h dec,L }

[0103] Additionally, an attention mechanism can be introduced to learn stylistic information from the text and encode the vector {h}. enc,0 h enc,1 , ..., h enc,M After processing through the attention mechanism, it is compared with the decoded vector {h} dec,0 h dec,1 , ..., h dec,L } is cascaded to obtain image captions {Y1, Y2, ..., Y} corresponding to the target image in multiple emotional styles. L Among these, attention mechanisms include soft attention.

[0104] Still with Figure 3 For example, further, GRU units can also be used to extract image captions {Y1, Y2, ..., Y}. L The semantic vectors in} are used to obtain the semantic vector P corresponding to the image captions. I,S And through the PAD 3D emotion model, the semantic vector P I,S Mapping to the PAD 3D emotional space yields PAD values ​​corresponding to image captions with different emotional styles.

[0105] In some examples, a classifier can be used to perform stylized text classification on the generated image captions to determine whether the generated image captions match the style of the input target image, and a style label can be added.

[0106] Corresponding to the above embodiments, the image caption generation model in the image-based emotion recognition method can be trained in the following way: acquiring sample data, which includes sample images and corresponding reference image captions, as well as stylized text data, which includes stylized text and corresponding reference style tags; inputting the sample images into the image caption generation model to obtain the output image captions corresponding to the sample images; adjusting the output image captions based on the style indicator and the stylized text data; determining a first loss value between the adjusted output image captions and the reference image captions; and adjusting the image caption generation model based on the first loss value.

[0107] Existing datasets can be directly selected as sample data (i.e., training sets) for training the image captioning generation model. For example, the image dataset (MSCOCO) and the unpaired stylized text dataset (Sentiment140) can be used. MSCOCO contains images and their corresponding image captions, which can be used to train the image captioning generation model to generate standard caption descriptions. Sentiment140 contains sentiment analysis text, including text data with three sentiments: negative, neutral, and positive, which can be used to instruct the image captioning generation model to learn the stylized features in this dataset.

[0108] For example, sample data can be collected by researchers specifically to evoke emotions in users, and multiple professionals can objectively describe the collected images to obtain image captions with various emotional styles as reference image captions. Further, the collected sample data can be used to train an image caption generation model to generate image captions with various emotional styles; then, by extracting the semantic vectors of the generated image captions with various emotional styles, and based on the PAD 3D emotion model, the PAD value corresponding to the image caption can be obtained, thereby determining the user's emotional state. It is understandable that when the sample data is large enough, the accuracy of the image captions generated by the image caption generation model can be improved, thus enabling the identification of different emotions evoked by different users when viewing different images.

[0109] Figure 4 This is a flowchart illustrating an image-based emotion recognition method provided in another embodiment of this application. Figure 4 As shown, it includes a network model training part and a network model application part. The network model training part includes:

[0110] S401: Prepare sample data.

[0111] The sample data can be the image dataset and unpaired stylized text dataset in the above embodiments, or it can be collected sample data.

[0112] Figure 5 This is a schematic diagram illustrating the model training of an image-based emotion recognition method provided in an embodiment of this application. Figure 5 As shown, the trained model (EmoCap) comprises two tasks and three stages: image captioning generation and emotion state recognition; and semantic primitive sequence generation, stylized text generation (i.e., image captioning generation with multiple emotion styles), and emotion state recognition. Image captioning generation can be decomposed into semantic primitive sequence generation and stylized text generation. Optionally, in the emotion state recognition stage, a classification model is also trained to obtain the semantic vector of the generated image captions, thereby enabling emotion state recognition and the addition of style labels to the image captions.

[0113] in addition, Figure 5 correspond Figure 4 The network model training part is also crucial to this application. The following will combine... Figure 4 , Figure 5 The network model training process is described in detail in conjunction with the above embodiments.

[0114] S402: Configure the neural network to extract image features from the sample data and generate semantic primitive sequences.

[0115] Based on existing sample data, a neural network can be configured to extract image features from the images in the sample data and generate semantic primitive sequences, i.e., semantic vectors. Specifically, the neural network can use Inception V3 to extract image features and use GRU units to convert the extracted image features into semantic vectors {X1, X2, ..., X}. M },correspond Figure 5 The first layer of the network structure.

[0116] For example, it can be extracted separately. Figure 5 The first layer of the network structure is used for network model training, and the average cross-entropy loss function based on semantic vectors, as shown in Equation (1), is used to guide the training of the network model. When the loss value of Equation (1) is minimized and converged, the optimized first network model is obtained. Therefore, when an image is input into this first network model, a more accurate semantic vector {X1, X2, ..., X} can be obtained. M Formula (1) is shown below:

[0117]

[0118] in, Let represent the true value of the semantic vector at position i in the semantic vector, M represent the length of the true value of the semantic vector, ∑ is the summation symbol, log represents the logarithmic function, p represents the probability, and I represents the input image.

[0119] The average cross-entropy loss function shown in formula (1) can reflect the semantic representation information in the image learned by the first network model.

[0120] S403: Generate stylized text using attention mechanisms.

[0121] exist Figure 5 In the second layer of the network structure, GRU units can still be used as the backbone network for stylized text generation, i.e. image caption generation with multiple emotional styles. However, attention mechanism and style indicator are introduced at the same time to better enhance the learning of image captions with multiple emotional styles and obtain more accurate stylized text.

[0122] When introducing style indicators, a teacher-forcing mode can be adopted. In this mode, during the training of the network model, the output h(t) of the previous state x(t) is not used as the input of the next state x(t+1). Instead, the expected output or the corresponding previous term y(t) of the sample data is directly used as the input of the next state x(t+1). This can generate semantic vectors that conform to the semantic information of the image and image captions that conform to stylization features more accurately.

[0123] For example, it can also be extracted separately. Figure 5 The second layer of the network structure is used for network model training, and the average cross-entropy loss function based on stylized text is adopted, as shown in Equation (2), to guide the training of the network model. When the loss value of Equation (2) is minimized and converged, the optimized second network model is obtained. Then, when the semantic vector output by the first network model is input into the second network model, stylized text {Y1, Y2, ..., Y} with higher accuracy can be obtained. L Formula (2) is shown below:

[0124]

[0125] in, L represents the stylized text real value (or reference image caption) at position j in the stylized text, and L represents the length of the stylized text real value.

[0126] The average cross-entropy loss function shown in formula (2) can reflect the stylized text information learned by the second network model, corresponding to the first loss value in the above embodiment.

[0127] It is understood that the first network model and the second network model in the above embodiments together constitute the image caption generation model.

[0128] S404: Configure a neural network to perform stylized text classification and extract semantic vectors to map to the PAD three-dimensional sentiment space.

[0129] For example, setting up a neural network for stylized text classification can correspond to... Figure 5 The third layer of the network structure, for example, uses a classification model to classify the generated stylized text.

[0130] Specifically, a classification model is used to perform stylized text classification on generated image captions with various emotional styles, which can be used to determine whether the image captions conform to the style of the target image. The classification model can be trained as follows: acquire stylized text data; input the stylized text from the stylized text data into the classification model to obtain the output style label corresponding to the stylized text; determine a second loss value between the output style label and the reference style label in the stylized text data; adjust the classification model based on the second loss value. The classification model can be a GRU unit. Alternatively, the classification model can be directly optimized and trained using a common classifier to obtain the optimized classifier for use as the classification model.

[0131] For example, training alone Figure 5 In the third layer network structure, i.e., the classification model, the average cross-entropy loss function based on stylized text classification can be used, as shown in formula (3), to guide the training of the classification model. When the loss value of formula (3) is minimized and converged, the optimized classification model is obtained, and then the stylized text {Y1, Y2, ..., Y...} output by the second network model is used. L When input into this classification model, stylized text {Y1, Y2, ..., Y} can be obtained. L The corresponding style tag Z k Formula (3) is shown below:

[0132]

[0133] Where p(z=z) k ) represents the probability that the generated stylized text belongs to the k-th style class. The style type represents the style of the actual stylized text, and exp represents the exponential function.

[0134] The average cross-entropy loss function shown in formula (3) can reflect the stylized text type information learned by the classification model, corresponding to the second loss value in the above embodiment.

[0135] In some embodiments, the three network models (first network model, second network model, and classification model) in the above embodiments, as well as the loss functions corresponding to the network models, are independent of each other and can be trained separately; or the three network models can be directly combined together for network model training; or the first network model and the second network model can be combined together for training, while the classification model is trained separately, and so on.

[0136] Alternatively, the average cross-entropy loss function in the above embodiments can be replaced with other loss functions that are beneficial to network model optimization, depending on the actual situation. This is not limited here.

[0137] In addition, during training, the network model in the above embodiments can be configured with relevant parameters of the training dataset and various hyperparameters appearing in the network structure according to the actual situation.

[0138] In some examples, with Figure 5 For example, the classification model in the above embodiment can extract stylized text {Y1, Y2, ..., Y}. L The semantic vectors in} are used to obtain the semantic vector P corresponding to the stylized text. I,S And through the PAD 3D emotion model, the semantic vector P I,S Mapping to the PAD 3D emotion space, and based on the emotion dictionary, the PAD value of the caption corresponding to the stylized text is obtained.

[0139] S405: Set the loss function to guide the training of the network model.

[0140] Specifically, during model training, the process involves continuous iteration until the loss value is minimized and convergence is achieved, at which point training is complete, and the optimized network model is obtained. For details, please refer to the aforementioned implementation example regarding the average cross-entropy loss function; it will not be repeated here.

[0141] Understandably, the optimized network model will achieve better evaluation results than the unoptimized network model during testing.

[0142] Furthermore, the obtained optimized network model can be applied, corresponding to Figure 4 The application of the network model is described in the following section. Specifically, the image and its corresponding descriptive text are input into the optimized network model. Based on the input image, various emotional styles of image captions are generated, and PAD values ​​based on the image captions are obtained. Then, based on the descriptive text, PAD values ​​based on the descriptive text are obtained. The PAD values ​​based on the image captions and those based on the descriptive text are weighted to obtain the target PAD value. Based on the set emotion state recognition experiment, the target PAD value is compared with the reference PAD value based on a standard scale to obtain the emotion state corresponding to the target PAD value, thus completing the emotion state recognition.

[0143] In addition, the emotional state recognition results in the above embodiments can be compared with the emotional states assessed by professionals based on standard scales to verify the rationality and effectiveness of the network model constructed and trained in this application.

[0144] Based on the above embodiments, the method provided in this application, based on a network model and a PAD three-dimensional emotional space, can generate image captions and obtain PAD values ​​more quickly and efficiently, and then realize the recognition of emotional states based on PAD values, without the need for professional personnel, which can reduce the pressure on professionals.

[0145] For example, in the attention-based stylized text generation module, image captions with multiple emotional styles {Y1, Y2, ..., Y} are generated. L Afterwards, several evaluation metrics can be used, such as Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understanding for Gisting Evaluation (ROUGE), and Metric for Evaluation of Translation with Explicit Ordered (METEOR), to evaluate the generated image captions. The higher the values ​​of BLEU, METEOR, and ROUGE, the more semantic information the generated image captions retain.

[0146] In addition, the optimal model performance can be compared with and without each key module, such as whether a stylized text generation module is included, whether the Sentiment140 dataset is used to generate stylized image captions (i.e. image captions with multiple mood styles), etc. The impact on the generated image captions is still evaluated using the above evaluation metrics.

[0147] When comparing the image captions generated by different models, the specific comparison results are shown in Table 2:

[0148] Table 2

[0149] Model BLEU@1 BLEU@3 METEOR ROUGE@1 NIC 0.6241 0.1403 0.3939 0.5159 EmoCap-coco 0.5652 0.1279 0.3645 0.4817 EmoCap 0.5614 0.1253 0.3629 0.4764

[0150] The Neural Image Caption (NIC) model is trained only on the Convolutional Neural Network-Recurrent Neural Network (CNN-RNN) encoder-decoder (i.e., the encoding and decoding sub-modules in the above embodiments), and is a traditional encoder-decoder model. The Stylized Image Caption Generation Model (EmoCap) is an image caption generation model provided in this application, such as... Figure 5 As shown, the EmoCap-coco model is a model obtained by modifying the training dataset on the EmoCap framework and training it only on neutral-style text (such as the MSCOCO dataset).

[0151] In Table 2, @1 and @3 refer to the number of consecutive words. For example, BLEU@1 measures word-level accuracy, while higher-order BLEU measures the fluency of the generated image captions.

[0152] As shown in Table 2, compared to the traditional NIC model, the evaluation results of EmoCap and EmoCap-coco show a slight decrease. This is because the EmoCap architecture provided in this application adds a Seq2Seq module to the original CNN-RNN structure, making the network structure more complex. However, overall, the decrease is small, and it can be approximated that the image captions generated by the model provided in this application on the MSCOCO dataset still have good semantic relevance representation.

[0153] Furthermore, the evaluation results of EmoCap and EmoCap-coco on various metrics are very close. EmoCap-coco is a model trained with unpaired stylized text data, indicating that unpaired stylized text data has almost no impact on the semantic relevance representation of image captions.

[0154] In some examples, image-stylized text data pairs and stylized text-stylized text classification result data pairs can be collected as test sets to test the accuracy of the trained image captioning generation model and classification model.

[0155] For example, based on image-stylized text data pairs, images are input into an image captioning generation model to generate image captions with various emotional styles, i.e., stylized text. The generated stylized text is compared with the stylized text in the image-stylized text data pairs to obtain the test results of stylized text generation, which is a confusion matrix, as shown in Table 3:

[0156] Table 3

[0157] Actual value / Predicted value negative neutral positive negative 1256 1902 842 neutral 0 4000 0 positive 607 2344 1049

[0158] Based on the stylized text-stylized text classification result data pairs, the stylized text is input into the classification model to obtain the classification result. The classification result is then compared with the stylized text classification result data pairs to obtain the test result of stylized text classification, which is a confusion matrix, as shown in Table 4.

[0159] Table 4

[0160] Actual value / Predicted value negative neutral positive negative 1641 2 286 neutral 2 1962 1 positive 289 3 1657

[0161] As shown in Table 3, the image caption generation model can generate relatively ideal stylized image captions from the input image with an accuracy of 52.54%, i.e., (1256+4000+1049) / (4000+4000+4000)=52.54%.

[0162] As shown in Table 4, the probability of correctly classifying stylized text using the classification model is 90.05%, i.e., (1641+1962+1657) / (1929+1965+1947)=90.05%, which demonstrates a good classification performance.

[0163] In some examples, the method provided in this application can also identify a user's emotional state for certain special images, where special images refer to images that include the user's actions, expressions, etc. In this case, the target PAD value can be obtained directly using the PAD value corresponding to the image caption, without combining it with the PAD value corresponding to the descriptive text, thereby identifying the user's emotional state.

[0164] In summary, this application has at least the following advantages:

[0165] 1. While preserving semantic information (i.e. semantic vectors), a style indicator is introduced, which innovatively realizes a variety of stylization features, namely generating image captions with multiple emotional styles, which is more conducive to recognizing different emotional states of different users.

[0166] 2. The method of identifying users' emotional states based on the PAD three-dimensional emotion model, combined with image and text modalities, still has good validity compared with traditional standard scales for identifying emotional states.

[0167] 3. Introducing an attention-based stylized text generation module can better learn the stylized features of images, solve the problem of single stylized image captions, and also help to identify different emotional states of different users.

[0168] 4. It can be applied to image caption generation and related fields. While ensuring the semantic accuracy of image captions, it can also realize image caption descriptions in different styles.

[0169] 5. Using deep learning technology for emotion state recognition is fast and efficient, and does not require professional personnel, which can reduce the pressure on professionals.

[0170] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0171] Figure 6 This is a schematic diagram of an image-based emotion recognition device according to an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. Figure 6 As shown, the image-based emotion recognition device 60 includes: an acquisition module 61, a generation module 62, a first determination module 63, a second determination module 64, and a third determination module 65. Wherein:

[0172] The acquisition module 61 is used to acquire the target image and the target user's descriptive text for the target image. The target image is an image that can evoke emotions in the user.

[0173] The generation module 62 is used to input the target image into the image caption generation model to generate image captions, and obtain image captions with various emotional styles corresponding to the target image;

[0174] The first determining module 63 is used to determine the first PAD value corresponding to each emotional style of the image caption in multiple emotional style image captions; and to determine the second PAD value corresponding to the descriptive text.

[0175] The second determining module 64 is used to determine the target PAD value based on multiple first PAD values ​​and second PAD values;

[0176] The third determining module 65 is used to determine the emotional state of the target user based on the target PAD value.

[0177] In one possible implementation, the second determining module 64 may be specifically used to: perform weighted processing on multiple first PAD values ​​and second PAD values ​​based on a preset formula to obtain a target PAD value. The preset formula is:

[0178]

[0179] Among them, y i a represents the first PAD value corresponding to the image caption of the i-th mood style. i R0 represents the scale score corresponding to the i-th emotional style, N represents the total number of emotional styles, and A represents the normalization coefficient.

[0180] In one possible implementation, the first determining module 63 may be specifically used to: determine a first semantic vector corresponding to the image caption of the emotional style, the first semantic vector containing multiple first semantic elements; for each of the multiple first semantic elements, determine a first semantic similarity between the first semantic element and each second semantic element in the second semantic vector, the second semantic vector being determined based on an emotion dictionary, the emotion dictionary including multiple emotion phrases; among the second semantic elements, determine a first target semantic element whose first semantic similarity to the first semantic element satisfies a first set condition; and perform weighted processing on the first semantic similarity corresponding to the first target semantic element and the PAD value corresponding to the first target semantic element to obtain a first PAD value.

[0181] In one possible implementation, the first determining module 63 may further be used to: determine a third semantic vector corresponding to the descriptive text, the third semantic vector containing multiple third semantic elements; for each of the multiple third semantic elements, determine a second semantic similarity between the third semantic element and each second semantic element in the second semantic vector; among the second semantic elements, determine a second target semantic element whose second semantic similarity with the third semantic element satisfies a second set condition; and perform weighted processing on the second semantic similarity corresponding to the second target semantic element and the PAD value corresponding to the second target semantic element to obtain a second PAD value.

[0182] In one possible implementation, the image caption generation model in the image-based emotion recognition device may include a convolutional neural network and an attention-based stylized text generation module. The stylized text generation module includes a GRU unit, an encoding submodule, and a decoding submodule. Specifically, the generation module 62 in the image-based emotion recognition device may be used to: input the target image into the convolutional neural network for feature extraction to obtain image features corresponding to the target image; input the image features into the GRU unit for semantic extraction to obtain a fourth semantic vector; input the fourth semantic vector into the encoding submodule for encoding processing to obtain an encoded vector; input the encoded vector into the decoding submodule for decoding processing to obtain a decoded vector; and concatenate the encoded vector and the decoded vector processed by the attention mechanism to obtain image captions with multiple emotion styles corresponding to the target image.

[0183] In one possible implementation, the third determining module 65 may be specifically used to: determine a reference PAD value corresponding to the target PAD value based on the preset correspondence between pleasure value, activation value, and dominance value and PAD value, according to the pleasure value, activation value, and dominance value of the target PAD value; and determine the emotional state of the target user based on the reference PAD value.

[0184] In one possible implementation, the image captioning generation model in the image-based emotion recognition device can be trained as follows: acquiring sample data, including sample images and corresponding reference image captions, and stylized text data, including stylized text and corresponding reference style tags; inputting the sample images into the image captioning generation model to obtain output image captions corresponding to the sample images; adjusting the output image captions based on style indicators and stylized text data; determining a first loss value between the adjusted output image captions and the reference image captions; and adjusting the image captioning generation model based on the first loss value.

[0185] In one possible implementation, the image-based emotion recognition device further includes: using a classification model to perform stylized text classification on generated image captions with multiple emotion styles, to determine whether the image captions conform to the style of the target image. The classification model can be trained by: acquiring stylized text data; inputting the stylized text from the stylized text data into the classification model to obtain the output style label corresponding to the stylized text; determining a second loss value between the output style label and a reference style label from the stylized text data; and adjusting the classification model based on the second loss value.

[0186] The image-based emotion recognition device provided in this application is similar in principle and technical effect to the above embodiments. For details, please refer to the above embodiments, which will not be repeated here.

[0187] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 700 includes:

[0188] Processor 701, memory 702, communication interface 703 and system bus 704.

[0189] The memory 702 and the communication interface 703 are connected to the processor 701 via the system bus 704 and communicate with each other. The memory 702 is used to store computer execution instructions, the communication interface 703 is used to communicate with other devices, and the processor 701 is used to execute the computer execution instructions to perform the scheme of the above-described image-based emotion recognition method embodiment.

[0190] Specifically, processor 701 may include one or more processing units. For example, processor 701 may be a CPU, a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0191] The memory 702 can be used to store program instructions. The memory 702 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a given function (such as image processing functions), etc. The data storage area may store data created during the use of the electronic device 700 (such as text data), etc. Furthermore, the memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, Universal Flash Storage (UFS), etc. The processor 701 executes various functional applications and data processing of the electronic device 700 by running the program instructions stored in the memory 702.

[0192] Communication interface 703 can provide solutions for wireless communication, including 2G / 3G / 4G / 16G, applied to electronic device 700. Communication interface 703 can receive electromagnetic waves via an antenna, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. Communication interface 703 can also amplify the signal modulated by the modem processor and radiate it as electromagnetic waves via the antenna. In some embodiments, at least some functional modules of communication interface 703 can be housed in processor 701. In some embodiments, at least some functional modules of communication interface 703 and at least some modules of processor 701 can be housed in the same device.

[0193] The system bus 704 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus 704 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0194] It should be noted that the number of memory 702 and processor 701 is not limited in this embodiment; there can be one or more of them. Figure 7 The diagram illustrates an example; the memory 702 and processor 701 can be connected via wired or wireless means, such as a bus connection. In practical applications, the electronic device 700 can be various forms of computers or mobile terminals. Computers include, for example, laptops, desktop computers, workbenches, servers, blade servers, mainframe computers, etc.; mobile terminals include, for example, personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0195] The electronic device in this embodiment can be used to execute the technical solutions in the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0196] Those skilled in the art will understand that Figure 7 The electronic devices shown do not constitute a limitation on electronic devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0197] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the image-based emotion recognition method described above.

[0198] This application also provides a computer program product, including a computer program that, when executed, implements the image-based emotion recognition method described above.

[0199] This application also provides a chip for executing instructions, the chip being used to perform the image-based emotion recognition method as described in any of the above method embodiments.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0201] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0202] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.< / sos>

Claims

1. An image-based emotion recognition method, characterized by, The method comprises: obtaining a target image and a description text of the target image input by a user, the target image being an image that can arouse the user's emotion; inputting the target image into an image caption generation model to generate an image caption, to obtain a plurality of emotion style image captions corresponding to the target image; the image caption generation model comprises a convolutional neural network and a style text generation module based on an attention mechanism, and the style text generation module comprises a GRU unit, an encoding submodule and a decoding submodule; for each emotion style image caption in the plurality of emotion style image captions, determining a first semantic vector corresponding to the emotion style image caption, the first semantic vector comprising a plurality of first semantic elements; for each first semantic element in the plurality of first semantic elements, determining a first semantic similarity between the first semantic element and each second semantic element in a second semantic vector, the second semantic vector being determined based on an emotion dictionary, the emotion dictionary comprising a plurality of emotion word groups; among the second semantic elements, determining a first target semantic element that satisfies a first set condition in terms of the first semantic similarity with the first semantic element; and performing weighted processing on a first semantic similarity corresponding to the first target semantic element and a PAD value corresponding to the first target semantic element to obtain a first PAD value; determining a second PAD value corresponding to the description text; determining a target PAD value according to a plurality of first PAD values and the second PAD value; determining an emotional state of the target user according to the target PAD value.

2. The emotion recognition method of claim 1, wherein, The method of determining a target PAD value according to a plurality of first PAD values and the second PAD value comprises: performing weighted processing on a plurality of first PAD values and the second PAD value based on a preset formula to obtain a target PAD value; wherein the preset formula is: wherein, represents the first PAD value corresponding to the image caption of the i-th emotional style, represents the scale score corresponding to the i-th emotional style, represents the second PAD value, N represents the total number of emotional styles, and A represents a normalization coefficient. 3.The emotion recognition method of claim 1, wherein, The method of determining a second PAD value corresponding to the description text comprises: determining a third semantic vector corresponding to the description text, the third semantic vector comprising a plurality of third semantic elements; for each third semantic element in the plurality of third semantic elements, determining a second semantic similarity between the third semantic element and each second semantic element in a second semantic vector; among the second semantic elements, determining a second target semantic element that satisfies a second set condition in terms of the second semantic similarity with the third semantic element; performing weighted processing on a second semantic similarity corresponding to the second target semantic element and a PAD value corresponding to the second target semantic element to obtain a second PAD value.

4. The emotion recognition method of any one of claims 1 to 3, characterized in that, The method of inputting the target image into an image caption generation model to generate an image caption to obtain a plurality of emotion style image captions corresponding to the target image comprises: inputting the target image into a convolutional neural network to extract features to obtain image features corresponding to the target image; inputting the image features into a GRU unit to extract semantics to obtain a fourth semantic vector; inputting the fourth semantic vector into the encoding submodule to perform encoding processing to obtain an encoding vector; and inputting the encoding vector into the decoding submodule to perform decoding processing to obtain a plurality of emotion style image captions corresponding to the target image. input the encoding vector into the decoding submodule for decoding processing to obtain a decoding vector; concatenate the encoding vector processed by the attention mechanism and the decoding vector to obtain an image caption of a plurality of emotional styles corresponding to the target image.

5. The emotion recognition method of any one of claims 1 to 3, characterized in that, The image caption generation model is trained in the following manner: obtain sample data, the sample data including sample images and reference image captions corresponding to the sample images and stylized text data, the stylized text data including stylized texts and reference style labels corresponding to the stylized texts; input the sample images into the image caption generation model to obtain output image captions corresponding to the sample images; adjust the output image captions based on a style indicator and according to the stylized text data; determine a first loss value of the adjusted output image captions and the reference image captions; adjust the image caption generation model according to the first loss value.

6. The emotion recognition method of any one of claims 1 to 3, characterized in that, The generated image captions of a plurality of emotional styles are classified into stylized texts by using a classification model, so as to determine whether the image captions conform to the style of the target image. The classification model is trained in the following manner: obtain stylized text data; input the stylized texts in the stylized text data into the classification model to obtain output style labels corresponding to the stylized texts; determine a second loss value of the output style labels and reference style labels in the stylized text data; adjust the classification model according to the second loss value.

7. An image-based emotion recognition apparatus, characterized by comprising: comprise: an acquisition module configured to acquire a target image and a description text of the target image by a target user, the target image being an image that can cause a user to produce an emotion; a generation module configured to input the target image into an image caption generation model to generate an image caption, so as to obtain a plurality of image captions of emotional styles corresponding to the target image; the image caption generation model comprising a convolutional neural network and a stylized text generation module based on an attention mechanism, the stylized text generation module comprising a GRU unit, an encoding submodule and a decoding submodule; a first determination module configured to determine, for each image caption of an emotional style in the plurality of image captions of emotional styles, a first semantic vector corresponding to the image caption of the emotional style, the first semantic vector comprising a plurality of first semantic elements; for each first semantic element in the plurality of first semantic elements, determine a first semantic similarity between the first semantic element and each second semantic element in a second semantic vector, the second semantic vector being determined based on an emotion dictionary, the emotion dictionary comprising a plurality of emotional word groups; among the second semantic elements, determine a first target semantic element whose first semantic similarity with the first semantic element satisfies a first set condition; and perform weighted processing on a first semantic similarity corresponding to the first target semantic element and a PAD value corresponding to the first target semantic element to obtain a first PAD value; determine a second PAD value corresponding to the description text; and determine a second PAD value corresponding to the description text. A second determining module is configured to determine a target PAD value according to the multiple first PAD values and the second PAD value. A third determining module is configured to determine an emotional state of the target user according to the target PAD value.

8. An electronic device, comprising: The method comprises the following steps: A memory and a processor; The memory is configured to store program instructions; The processor is configured to invoke the program instructions to execute the image-based emotion recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed to implement the image-based emotion recognition method according to any one of claims 1 to 6.

Images