An emotional image description method and system based on reinforcement learning

By extracting the theme color characteristics and emotional word embedding library of the image, combined with reinforcement learning technology, and optimizing image description, the problem of insufficient emotion and color expression in image description is solved, and more accurate and vivid emotional image description is achieved.

CN114639139BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210151256.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-16
Publication Date
2024-11-08
Estimated Expiration
2042-02-16

AI Technical Summary

Technical Problem

The prior art lacks effective extraction and expression of image emotions and colors in image description, which makes it difficult for people with visual impairment and cognitive inadequate understanding to obtain emotional information from images, and the generated description sentences are stiff and not very abstract, and cannot be accurately conveyed. The mood and atmosphere of the image.

Method used

By constructing an emotional image description method based on reinforcement learning, the theme color characteristics of the image are extracted and embedded into the library of emotions words to generate initial emotional descriptions, and the description is optimized using reinforcement learning fine-tuning module to make it more vivid and emotional.

Benefits of technology

Effectively identify and describe multiple emotions in the image, improve the accuracy and robustness of image description, make the generated description statements more vivid and emotional, and can better convey the emotional information of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114639139B_ABST
    Figure CN114639139B_ABST
Patent Text Reader

Abstract

The present invention discloses an emotional image description method based on reinforcement learning, which relates to the technical field of image processing and pattern recognition. An emotional word embedding library is constructed based on a large-scale corpus; an image emotion recognition model is constructed; the image emotion recognition model is trained using an image emotion analysis data set; an image factual description model is constructed; the image factual description model is trained using an image description data set; an emotional image description initialization module is constructed, and an initial emotional image description is generated using the emotional word embedding library, the image emotion category output by the image emotion recognition model, and the image factual description output by the image factual description model; a fine-tuning module based on reinforcement learning is constructed to fine-tune the initial emotional image description to generate a final emotional image description. The present invention also discloses an emotional image description system based on reinforcement learning, which can make the image descriptions of various complex scenes more vivid and emotional.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and pattern recognition, and in particular to an emotional image description method and system based on reinforcement learning. Background Art

[0002] Today's information society is full of image data, such as daily life photos, medical images and remote sensing satellite images. People's way of information interaction has changed from traditional voice and text to multimodal information interaction. Image information interaction is an important core of the future "metaverse", but for people with visual impairments and cognitive deficiencies, they cannot accurately obtain information from images, including image semantic information and emotional information, which seriously affects the smoothness of information interaction and the convenience of information acquisition for such groups in the future.

[0003] Emotion is an important part of information interaction. Both parties need to interact emotionally to obtain the other party's emotions and complete better information exchange. For people with visual impairments and cognitive deficiencies, a certain degree of factual image description can help them understand the object of the image. Since factual description simply describes the objects contained in the image and lacks the expression of emotions and colors in the image, people cannot obtain the emotions conveyed by the image. Therefore, emotional image description has become the most challenging problem in image description. The color of the image is input into the computer through the pixel value obtained by RGB, but people do not have any feelings or associations with specific values. Therefore, directly extracting the pixel value size cannot allow people with visual impairments and cognitive deficiencies to intuitively feel the various types of information contained in the image. The color information contained in the image is diverse, and there is a certain image artistic conception. At the same time, the emotions contained in the image are ever-changing, including 8 types of emotions: amusement, anger, surprise, acceptance, hatred, ecstasy, fear, and sadness. Therefore, according to the emotion analysis results of the image emotion recognition model and the generated image factual description semantics, emotion words corresponding to the image emotion category are selected from the emotion word embedding library to generate the initial emotional image description; finally, a fine-tuning module based on reinforcement learning is constructed. This module uses the reinforcement learning method to fine-tune the generated initial emotional image description to generate the final emotional image description, making the semantics more fluent and rich.

[0004] At present, the research on image description is mostly focused on single factual image description. Although the generated image description has simple syntax and the model is highly interpretable, the nonlinear mapping relationship between image features and image text description is simple, resulting in low abstraction and limited expression ability. In addition, the generated description is relatively stiff and lacks emotional color, and it is impossible to describe the content of the image in more detail through the generated text. At the same time, the single factual image description lacks description of the objects in the image and their interactive relationships. Furthermore, the factual image description lacks description of the emotions conveyed by the image color and cannot fully describe the color atmosphere information shown by the image.

[0005] The Chinese patent application "A method for generating image description based on deep attention mechanism" (patent application number 201711073398.1, publication number CN108052512B) constructs a deep long short-term memory network model, adds an attention mechanism function between the units of the long short-term memory network model, and uses the training picture features extracted by the convolutional neural network and the description information of the training pictures to train the long short-term memory network with the attention mechanism function, so as to obtain a deep long short-term memory network model; in the image description generation step, the image to be described is passed through the convolutional neural network model and the deep long short-term memory network model in turn to generate a description corresponding to the image. The problem with this method is that the image to be described is directly encoded and decoded using the long short-term memory network model to obtain the image description, and the internal features of the image are not fully extracted, and there is no emotion extraction based on the theme color features of the image, which may affect the semantic richness of the final image emotion recognition, and there is no emotional support. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an emotional image description method and system based on reinforcement learning. On the one hand, the theme color feature priors of the training set images are extracted and input into the image emotion recognition model together with the training set images, the network model parameters are optimized, and the image factual description model with multi-feature information fusion is combined to generate the initial emotional image description; on the other hand, the initial emotional image description is fine-tuned through reinforcement learning to make the sentence more fluent and emotional; the complementary role between image emotion recognition and image semantic description is further brought into play, and the accuracy and robustness of the image description are improved while obtaining the emotional image description.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] An emotional image description method based on reinforcement learning proposed in the present invention comprises the following steps:

[0009] Step 1: Build an emotional word embedding library based on a large-scale corpus;

[0010] Step 2: Build an image emotion recognition model;

[0011] Step 3: Use the image emotion analysis dataset to train the image emotion recognition model;

[0012] Step 4: construct an image factual description model based on an attention mechanism for generating image factual descriptions, wherein the image factual description model includes an image factual description preprocessing module, an image feature encoder, and a feature-text decoder which are sequentially connected;

[0013] Step 5: Use the image description dataset to train the image factual description model;

[0014] Step 6: construct an emotional image description initialization module. The emotional image description initialization module selects emotional words corresponding to the image emotion category from the emotional word embedding library according to the image emotion category output by the trained image emotion recognition model, and embeds them into the image factual description output by the trained image factual description model to generate an initial emotional image description;

[0015] Step 7: Construct a fine-tuning module based on reinforcement learning. The fine-tuning module is used to fine-tune the initial emotional image description to generate the final emotional image description.

[0016] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, in step seven, the fine-tuning module based on reinforcement learning includes a sentence reconstruction generator, a sentence storage unit, a sentence sampling unit, a sentence evaluation unit and a word selection evaluation unit; wherein, the sentence reconstruction generator serves as an intelligent agent in the reinforcement learning system, and the sentence storage unit, the sentence sampling unit, the sentence evaluation unit and the word selection evaluation unit constitute the external environment in the reinforcement learning system; the sentence reconstruction generator continuously interacts with the external environment, obtains the reward information of the external environment, learns the mapping from the environment state to the behavioral action, to optimize and adjust the behavioral action, fine-tune the initial emotional image description, and generate the final emotional image description.

[0017] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, in step seven, the fine-tuning module based on reinforcement learning includes a sentence reconstruction generator, a sentence storage unit, a sentence sampling unit, a sentence evaluation unit and a word selection evaluation unit. The specific method for the fine-tuning module to fine-tune the initial emotional image description is as follows:

[0018] Step 701: The sentence reconstruction generator selects words with similar semantics from the sentiment word embedding library through the word selector according to the environmental state at time t-1 and the reward at time t-1, performs the word selection action, and adds the selected words to the sentence S generated at time t-1. t-1 In the example above, the statement S at time t is generated. t ; Among them, the statement S0 generated at time 0 is the statement generation start symbol, and the environment state at time t-1 is the statement S generated at time t-1 t-1 , the reward R at time t-1 t-1 That is, the score of the word selected at time t-1, where t is the time;

[0019] Step 702: The statement storage unit stores the updated statement S at time t. t The sentence sampling unit updates the sentence S at the tth moment based on the sampling search algorithm. t Roll back to generate N sentences, where N is 3, 4 or 5; the sentence evaluation unit first evaluates and scores the N sentences generated by the sentence sampling unit using the emotion identifier, grammatical collocation identifier and semantic identifier to obtain N emotion reward scores, grammatical collocation reward scores and semantic reward scores, and then takes the weighted average method to obtain the comprehensive reward score, and finally inputs the comprehensive reward score into the word selection evaluation unit; the word selection evaluation unit outputs the score of the selected word as the reward R fed back by the external environment to the sentence reconstruction generator t ;

[0020] Step 703, iterate step 701 to step 702, the sentence reconstruction generator continuously interacts with the external environment until the maximum reward for sentence reconstruction is obtained, and the final emotional image description is generated.

[0021] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, the sampling search algorithm of the sentence sampling unit adopts polynomial sampling or Monte Carlo sampling method.

[0022] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, in step 1, the specific method of constructing the emotional word embedding library is as follows:

[0023] Step 101: Use the NLTK tool to obtain nouns and verbs in the target detection and image description datasets, generate a semantic vocabulary, and calculate the word vector of each semantic word therein;

[0024] Step 102: Filter out emotion words from the large-scale corpus LSCC, generate an emotion word library, and calculate the emotion word vector of each emotion word; classify the emotion words corresponding to each semantic word in the semantic word library into 8 categories defined by IAPS: pleasure, rage, surprise, acceptance, hatred, ecstasy, fear, and grief;

[0025] Step 103: Filter out emotion phrases of different emotion categories corresponding to semantic words from the emotion word library, and construct an emotion word embedding library.

[0026] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, the image emotion recognition model includes an image emotion recognition preprocessing module, a facial emotion feature extraction module, an image theme color feature extraction module, an image emotion feature extraction module, a feature fusion layer, a fully connected layer and a classification layer; the image emotion recognition preprocessing module includes a face detection unit, a face image normalization processing unit and an image size normalization processing unit; wherein,

[0027] The face detection unit detects the face area in the input image using a pre-trained face detection network and labels different face areas;

[0028] The face image normalization processing unit is used to crop, align and normalize the size of each detected face area;

[0029] The image size normalization processing unit is used to normalize the size of the input image;

[0030] The facial emotion feature extraction module is used to extract facial emotion features of each person in the cropped, aligned and size-normalized facial image;

[0031] The image theme color feature extraction module is used to extract the theme color features of the input image;

[0032] The image emotion feature extraction module is used to extract the emotion feature of the size-normalized image output by the image size normalization processing unit;

[0033] The feature fusion layer is used to fuse the facial emotion features output by the face emotion feature extraction module, the theme color features output by the image theme color feature extraction module, and the emotion features output by the image emotion feature extraction module to obtain a fused emotion feature vector;

[0034] The fully connected layer is used to fully connect the feature fusion layer and the classification layer;

[0035] The classification layer is used to output the emotion category to which the image belongs.

[0036] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, in step 2, the specific method of using the image theme color feature extraction module to extract the theme color feature of the input image is as follows:

[0037] Step 1: Use the microelement method to cut the RGB space into independent three-dimensional blocks;

[0038] Step 2: Put the RGB format pixel scattered points of the image into the cut RGB space, and use the scattered point value as the value of the three-dimensional block. If there is no scattered point in the three-dimensional block, use the center value of the three-dimensional block area as the value of the three-dimensional block.

[0039] Step 3: weighted summing the values ​​of the three-dimensional blocks in the entire sliding window area is performed by weighted sliding window method to obtain the value of the three-dimensional block of the sliding window size. The size of the sliding window depends on the number of types of image theme colors to be finally selected.

[0040] Step 4: Through steps 1 to 3, the image theme color features of the input image are finally obtained.

[0041] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, the microelement method is used to cut the RGB space to form independent three-dimensional blocks, and the three-dimensional blocks are cubic blocks of pixel size.

[0042] As a further optimization scheme of the emotional image description method based on reinforcement learning described in the present invention, the specific method of constructing an image factual description model in step 4 is as follows:

[0043] Step 4.1, image factual description preprocessing module, using the network model pre-trained on the target detection and target relationship detection data set to preprocess the input image; the specific method is as follows: 1) through the pre-trained target detection algorithm, detect the areas where various types of targets appearing in the image are located; using the pre-trained target relationship detection algorithm, detect the areas where various types of target interactions appearing in the image; 2) cropping and aligning the input image, the images of the areas where various types of targets are located, and the images of the areas where various types of targets interact are located, and normalizing them to obtain the normalized image of the input image, the normalized image of the areas where various types of targets are located, and the normalized image of the areas where various types of targets interact;

[0044] Step 4.2, construct an image feature encoder, which includes an image global feature encoding branch, a target feature encoding branch, an inter-target interaction feature encoding branch, an attention mechanism and a feature fusion layer; the image global feature encoding branch includes a plurality of convolution modules, the input of the image global feature encoding branch is a normalized image of the input image, which is used to extract the global features of the image and convert it into a vector form; the target feature encoding branch includes a plurality of convolution modules, the input of the target feature encoding branch is a normalized image of the area where various types of targets are located, which is used to extract local target features and convert them into vector form; the inter-target interaction feature encoding branch includes a plurality of convolution modules, the inter-target interaction feature encoding branch The input of the branch is a normalized image of the area where various types of target interactions are located, which is used to extract the features of the action interaction area between targets and convert them into vector form; the convolution module includes one or more convolution layers and a pooling layer; the attention mechanism is used to capture the target features and target interaction features that need to be focused on relative to the global features; the feature fusion layer is used to normalize the above-mentioned image global features, the focused target features and the focused target interaction features, respectively, and then splice them through the fully connected layer to output an image feature vector; the fully connected layer fully connects the output of the feature fusion layer to the c output neurons of this layer, and outputs a c-dimensional feature vector;

[0045] Step 4.3, construct a feature-text decoder, wherein the input of the feature-text decoder is the image feature vector obtained by processing the image feature encoder; and use a combination module including at least 2 layers of long short-term memory (LSTM) networks to decode the image feature vector into text.

[0046] An emotional image description system based on reinforcement learning, comprising:

[0047] Emotional word embedding library, which is built on the basis of a large-scale corpus to provide corpus support for the final generation of emotional image description;

[0048] The image emotion recognition model comprises an image emotion recognition preprocessing module, a face emotion feature extraction module, an image theme color feature extraction module, an image emotion feature extraction module, a feature fusion layer, a fully connected layer and a classification layer; the image emotion recognition preprocessing module comprises a face detection unit, a face image normalization processing unit and an image size normalization processing unit; wherein the face detection unit detects the face area in the input image by using a pre-trained face detection network and labels different face areas; the image size normalization processing unit is used to normalize the input image. The pixel size of the input image is normalized to obtain a unified image input size; the face image normalization processing unit is used to crop, align and normalize the size of each detected face area; the face emotion feature extraction module includes multiple convolution modules; the image theme color feature extraction module uses a color clustering method to extract theme color features from the training set images in the image emotion analysis data set; the fully connected layer is used to fully connect the feature fusion layer and the classification layer; the classification layer is used to output the emotion category to which the image belongs; finally, the image emotion analysis data set is used to train the image emotion recognition model;

[0049] The image factual description model includes an image factual description preprocessing module, an image feature encoder and a feature-text decoder; the image factual description preprocessing module preprocesses the input image; the image feature encoder includes an image global feature encoding branch, a target feature encoding branch, an inter-target interaction feature encoding branch, an attention mechanism and a feature fusion layer; the image global feature encoding branch is used to extract the global features of the image and convert them into a vector form; the target feature encoding branch is used to extract local target features and convert them into a vector form; the inter-target interaction feature encoding branch is used to extract The action interaction area features are converted into vector form; the attention mechanism is used to capture the target features and target interaction features that need to be focused on relative to the global features; the feature fusion layer is used to normalize the above-mentioned image global features, target features that are focused on, and target interaction features that are focused on, respectively, and then output an image feature vector through splicing through a fully connected layer; the input of the feature-text decoder is the image feature vector obtained by the image feature encoder; and the image feature vector is decoded into text using the combination module of the long short-term memory LSTM network; finally, the image description dataset is used to train the image factual description model;

[0050] The emotional image description initialization module selects the emotional words corresponding to the image emotion category from the emotional word embedding library according to the image emotion output by the trained image emotion recognition model, and embeds them into the image factual description output by the trained image factual description model to generate the initial emotional image description;

[0051] A fine-tuning module based on reinforcement learning uses reinforcement learning to adjust the generated initial emotional image description; the reinforcement learning fine-tuning module includes a sentence reconstruction generator, a sentence storage unit, a sentence sampling unit, a sentence evaluation unit and a word selection evaluation unit; wherein the sentence reconstruction generator serves as an intelligent agent in the reinforcement learning system, and the sentence storage unit, the sentence sampling unit, the sentence evaluation unit and the word selection evaluation unit constitute the external environment in the reinforcement learning system; the sentence reconstruction generator continuously interacts with the external environment, obtains the reward information of the external environment, learns the mapping from the environment state to the behavior action, optimizes and adjusts the behavior action, fine-tunes the initial emotional image description, and generates the final emotional image description.

[0052] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0053] The present invention can effectively identify eight image emotions, namely, amusement, anger, awe, contentment, disgust, excitement, fear, and sadness, by introducing the subject color feature prior into image emotion recognition. The present invention selects emotion words corresponding to the image emotion categories from the emotion word embedding library, embeds them into the factual description of the image, generates an initial emotional image description, and uses a reinforcement learning method to fine-tune the generated initial emotional image description, so that the generated emotional image description is more vivid and emotional. The details are as follows:

[0054] (1) The existing emotion analysis methods mainly divide emotions into positive emotions, negative emotions and neutral emotions; in the present invention, eight image emotions, namely, amusement, anger, awe, contentment, disgust, excitement, fear and sadness, are used to classify emotion categories, which belongs to fine-grained image emotion analysis.

[0055] (2) The corpus in the present invention is designed from large-scale target detection datasets, image description datasets, and sentiment analysis datasets, and adopts the method of spatial classification of emotional words, which can avoid the defects of semantic similarity but large emotional gap. At the same time, it can learn vocabulary collocation from a large amount of annotated corpus, so that better image description results can be obtained.

[0056] (3) Compared with the existing image emotion analysis model, the image emotion recognition model in the present invention gives the image theme color feature prior through the color clustering method, and adopts a dual-branch network model based on the facial emotion feature extraction module and the image emotion feature extraction module to respectively extract features of the facial image after cropping, alignment and size normalization of the input image and the size normalization of the global image output by the image size normalization processing unit; so that the image emotion recognition model can not only obtain the facial emotion category of the face but also fully capture the global emotion information of the image, promote the information interaction between the facial emotions of the face and the emotions of the overall atmosphere of the image, and have stronger representation and generalization capabilities.

[0057] (4) In the process of factual description of an image, the present invention preprocesses the target area features, target features, and target relationship features that are focused on in the image, and uses the attention mechanism to judge the focused targets and their corresponding relationships, so that the features of the input image factual description model are richer and more closely related to the input image content, making the image factual description more reasonable.

[0058] (5) In the process of generating the initial emotional description, the present invention uses the image emotion recognition results to determine the image emotion words, and combines the generated image factual description semantics to generate the initial emotional image description. These features are obtained by extracting the features of the image itself. Therefore, the obtained initial emotional image description is highly correlated with the original image semantics and has stronger representation and generalization capabilities.

[0059] (6) The present invention uses a reinforcement learning method to fine-tune the generated initial emotional image description, while maintaining the original semantics, solving the problems of unreasonable semantic matching and inaccurate application of emotional vocabulary. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of the steps of the emotional image description method based on reinforcement learning of the present invention.

[0061] Figure 2 It is a structural diagram of the emotional image description system based on reinforcement learning of the present invention.

[0062] Figure 3 is a structural diagram of the image emotion recognition model used in the embodiment of the present invention.

[0063] Figure 4 1 is an example diagram of the theme color feature of an image in an embodiment of the present invention; wherein (a) is the theme color feature of the image obtained by solution, and (b) is the scatter point mapping of the image in the RGB space.

[0064] Figure 5 It is a structural diagram of the image factual description model used in the embodiment of the present invention.

[0065] Figure 6 It is a structural diagram of the fine-tuning module based on reinforcement learning of the present invention.

[0066] Figure 7 This is an example of an image from the Ai Challenger Caption 2017 database. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] The present invention aims at the problem that the image description network uses an encoder to directly extract the global features of the image, and uses a decoder to directly map the text to generate the image description, but the text description is semantically deficient and lacks emotional expression. The purpose of the present invention is to provide an emotional image description method and system based on reinforcement learning to solve the problem that the existing technology cannot accurately and vividly describe the image, and to provide a more vivid and emotional image description system for infant education and guidance and visually impaired people. A new way and method is opened up. The development of an emotional image recognition system provides a more vivid and emotional image description for the visually impaired and infant auxiliary education, which is of great significance and value for the visually impaired to more vividly understand the content expressed by the image and the emotions it conveys.

[0069] like Figure 1 As shown, the emotional image description method based on reinforcement learning provided by the embodiment of the present invention mainly includes the following steps:

[0070] Step 1: Build an emotional word embedding library based on a large-scale corpus. First, use the NLTK (Natural Language Toolkit) tool to obtain nouns and verbs in the target detection and image description datasets to generate a semantic word library. Then, filter out emotional words from the large-scale corpus LSCC (Large Scale Chinese Corpus) to generate an emotional word library. Finally, calculate the emotional word corresponding to each semantic word in the semantic word library to build an emotional word embedding library.

[0071] In this embodiment, the NLTK tool is used to obtain nouns and verbs in the object detection dataset COCO and the image description dataset MSCOCO and flickr30k, and a semantic vocabulary C-corpus = {N, V} is generated, which includes a noun library N = {N i |i=1,2…n1}, where n1 represents the number of object types that the algorithm can recognize; verb library V={V i |i=1,2…n2}, where n2 represents the number of action categories that can be detected by relation detection. And calculate the word vector of each word;

[0072] The emotion words are selected from the large-scale corpus - NRC emotion corpus, and the emotion word library S-corpus = {ADJ, ADV} is generated, which includes the adjective library ADJ = {ADJ i |i=1,2…m1}, m1 indicates the number of optional adjectives; adverb ADJ={ADJ i |i=1,…m2}, m2 represents the number of optional adverbs. The S-corpus is divided into three categories (positive, negative, neutral) and subdivided into eight categories defined by IAPS (International Affective Picture System): Amusement, Anger, Awe, Contentment, Disgust, Excitement, Fear, Sadness; each category is represented by c i ={c1,c2,…,c8}; The specific classification method is as follows: the sum of the distances between the emotional word embedding vector after the semantic word vector and the emotional word vector in each group are concatenated and the 8 benchmark emotional word vectors is used as the objective function. By minimizing the objective function, the spatial classification of the emotional word embedding vector is solved; let the objective function of the classification be where u i c i The centroid of the class, x ij is the emotional word embedding vector after the fusion of the i-th semantic word and the j-th emotional word;

[0073] Construct the emotional strength relationship between each semantic word and the emotional word. The strength of the relationship is determined by the emotional classification probability detected by the text emotion recognition algorithm BERT.

[0074] The final constructed emotional vocabulary is shown in Table 1;

[0075] Table 1. Examples of emotional vocabulary

[0076]

[0077] The constructed sentiment word embedding library is shown in Table 2;

[0078] Table 2 Examples of sentiment word embedding library

[0079]

[0080] Step 2: Construct a Figure 3 The image emotion recognition model shown in the figure comprises an image emotion recognition preprocessing module, a facial emotion feature extraction module including at least two convolution modules, an image theme color feature extraction module, an image emotion feature extraction module including at least two convolution modules, a feature fusion layer, a fully connected layer and a classification layer; the convolution module includes at least one convolution layer and one pooling layer;

[0081] The image emotion recognition preprocessing module includes face detection, face image normalization processing, and image size normalization processing; the face detection uses a pre-trained face detection network to detect the face area in the input image and label different face areas; the face image normalization processing crops and aligns each detected face area and normalizes each processed face image; the image size normalization processing is used to normalize the input image; the image size normalization processing is used to normalize the input image to obtain a unified image input size;

[0082] The facial emotion feature extraction module includes a plurality of convolution modules, the input of which is the facial expression image output by the image emotion recognition preprocessing module, and is used to extract the facial emotion features of a person;

[0083] The image theme color feature extraction module uses a color clustering method to extract theme color features from the training set images in the image emotion analysis data set, and encodes the image theme color features into vectors through a word embedding method as prior knowledge of the image emotion;

[0084] The image emotion feature extraction module is used to extract image emotion features, and includes multiple convolution modules. The input of the module is the size-normalized image output by the image size normalization processing unit in the image emotion recognition preprocessing module; the convolution module includes one or more convolution layers and a pooling layer.

[0085] The convolution module includes one or more convolution layers and a pooling layer;

[0086] The feature fusion layer is used to fuse the facial emotion features output by the face emotion feature extraction module, the theme color features output by the image theme color feature extraction module, and the emotion features output by the image emotion feature extraction module to obtain a fused emotion feature vector;

[0087] The fully connected layer is used to fully connect the feature fusion layer and the classification layer; the classification layer is used to output the emotion category to which the image belongs;

[0088] This embodiment constructs an image emotion recognition model, such as Figure 3 As shown, the specific implementation is as follows:

[0089] (1) Image emotion recognition preprocessing module, including face detection, face image normalization processing, image theme color feature extraction module, and image size normalization processing;

[0090] Face detection, first use the pre-trained FaceNet network to detect the input image Figure 7 (Image source: AiChallenger Caption 2017; Image ID: 1059397860565088790) The face area is then cut out, and finally the different face areas cut out in the image are numbered;

[0091] Normalization of face images, for normalizing different face regions obtained by the above face detection to obtain an image with a size of 56×56 pixels;

[0092] Image size normalization is used to normalize the input image to an image of 224×224 pixels;

[0093] (2) Figure 4 As shown in (b) of the figure, the image theme color feature extraction module first maps the images in the MSCOCO dataset into scattered points in the RGB space, and extracts the theme color of the image through the color clustering algorithm to obtain the following Figure 4 The result shown in (a) in FIG. 1 is obtained; then, the color clustering result is converted into HSV (hue, saturation, brightness) format; finally, the image theme color feature is encoded into a vector through the word embedding method as the prior knowledge of the image emotion; in this embodiment, the method based on the direct clustering of traditional RGB scatter points will cause the scatter point colors to be neutralized, resulting in two types of obviously different color scatter points being clustered into a color category that is obviously different from itself, so the clustering method of color scatter points is modified:

[0094] First, the RGB space is cut into pixel-sized cubes using the microelement method. Then, the scattered points of the image's RGB space are placed in the cut RGB space, and the scattered point values ​​are used as the values ​​of the cubes. If there are no scattered points in the cube, the center value of the cube area is used as the value of the cube.

[0095] Then, a three-dimensional sliding window method is used to cluster the three-dimensional blocks into three-dimensional blocks of the size of the sliding window, and the three-dimensional block values ​​in the entire sliding window area are weighted and summed up by the sliding window weighting method to obtain the value of the three-dimensional block of the size of the sliding window. The size of the sliding window depends on the number of types of image theme colors to be selected in the end. The window weight is processed by the rotation and translation method in order to make the color transition more reasonable and smooth. The specific method is as follows: (1) If the RGB scatter point values ​​of the original image are distributed relatively evenly in the window area, the weight is assigned by decreasing from the center of the window to the surrounding area; (2) If the RGB scatter point values ​​of the original image are distributed in the corners or edge areas of the window, Then the weight window that decreases from the center of the window to the surrounding areas is gradually moved toward the direction with more scattered points in the window, so as to make the center of the window close to the original RGB scattered point clustering area in the area as much as possible; (3) For the weight in the window that exceeds the weight of this part due to sliding, the weight above the window is rotated. If there are multiple clustering points in the window area, this method is used to slide multiple times and weighted average. The weight size depends on the original RGB scattered point clustering degree. The image theme color feature results (RGB form) are shown in Table 3: orange (6.40%), dark green (17.8%), light gray (18.6%), olive (23.2%), brown (34.0%);

[0096] Table 3. Examples of image emotion recognition results

[0097]

[0098] Finally, the extracted image theme color is converted into HSV format. The image theme color is converted into HSV (hue, saturation, brightness) format. First, the RGB result is preprocessed. The preprocessing formula is as follows:

[0099] R′=R / 255; G′=G / 255; B′=B / 255; C max =max(R′,G′,B′)

[0100] C min =min(R′,G′,B′); Δ=C max -C min

[0101] The conversion formula of saturation S is as follows:

[0102]

[0103] Brightness V is expressed as V = C max ; The calculation formula of hue H is as follows:

[0104]

[0105] The output is the HSV features in Table 3: hue 11.59, saturation 0.36, brightness 0.63; and the HSV features are encoded into a 1024-dimensional vector;

[0106] (3) The facial emotion feature extraction module includes three sequentially connected convolution modules and a feature fusion layer, and is specifically implemented as follows:

[0107] Convolution module d1: includes 2 convolution layers and 1 pooling layer. Both convolution layers use 128 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 56×56×128. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 28×28×128.

[0108] Convolution module d2: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 256 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 28×28×256. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 14×14×256.

[0109] Convolution module d3: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 512 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps with a size of 14×14×512. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps with a size of 7×7×512.

[0110] The feature fusion layer c1 takes as input the facial emotion features output by different emotional feature branches in the facial emotion feature extraction module, both of which have a size of 7×7×512. The two feature maps are subjected to global average pooling operations to obtain two 512-dimensional feature vectors, which are then fused and finally output as a 512-dimensional feature vector.

[0111] (4) The image emotion feature extraction module includes five convolution modules connected sequentially, as follows:

[0112] Convolution module d4: includes 2 convolution layers and 1 pooling layer. Both convolution layers use 64 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 224×224×64. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 112×112×64.

[0113] Convolution module d5: includes 2 convolution layers and 1 pooling layer. Both convolution layers use 128 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping. The output size is 112×112×128 feature maps. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2. The output size is 56×56×128 feature maps.

[0114] Convolution module d6: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 256 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 56×56×256. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 28×28×256.

[0115] Convolution module d7: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 512 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 28×28×512. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 14×14×512.

[0116] Convolution module d8: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 512 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps with a size of 14×14×512. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps with a size of 7×7×512.

[0117] (5) Feature fusion layer c2, which takes as input the image emotion feature of size 7×7×512 output by the image emotion feature extraction module and the 512-dimensional facial emotion feature output by the feature fusion layer c1. It performs global average pooling on the image emotion feature to obtain two 512-dimensional feature vectors, which are then fused with the 512-dimensional facial emotion feature. Finally, the two fused feature vectors are concatenated to output a 1024-dimensional feature vector. The 1024-dimensional feature vector is fused with the 1024-dimensional image theme color feature output by the image theme color feature extraction module to obtain a new 1024-dimensional vector.

[0118] (6) Fully connected layer b1, containing 256 neurons, used to fully connect the feature fusion layer and the classification layer;

[0119] (7) Classification layer a1, which uses a Softmax classifier and contains 8 neurons, outputs the emotion category to which the image belongs;

[0120] Step 3: Use the image emotion analysis dataset ArtPhoto to train the image emotion recognition model;

[0121] This embodiment uses the ArtPhoto image emotion analysis dataset. The ArtPhoto image emotion analysis dataset uses three datasets: images in IAPS, ArtPhoto, and Abstract Paintings, which contain a total of 1429 image samples, each sample corresponding to an expression category, including 8 emotion categories: Amusement, Anger, Awe, Contentment, Disgust, Excitement, Fear, and Sadness. In practice, other image emotion analysis datasets can also be used, or image emotion analysis datasets can be collected by themselves to establish an image emotion analysis dataset containing emotion category labels.

[0122] Step 4: Construct a Figure 5 The image factual description model based on the attention mechanism shown in the figure includes an image factual description preprocessing module, an image feature encoder and a feature-text decoder;

[0123] The image factual description preprocessing module uses a network model pre-trained on a target detection and target relationship detection data set to preprocess the input image; the specific method is as follows: 1) using a pre-trained target detection algorithm, detecting the areas where various targets appearing in the image are located, cropping and aligning the areas where various targets appearing in the image are located, and normalizing the processed images of the areas where various targets appearing in the image are located; 2) using a pre-trained target relationship detection algorithm, detecting the areas where various targets appearing in the image are interacting, cropping and aligning the areas where various targets appearing in the image are interacting, and normalizing the processed images of the areas where various targets appearing in the image are interacting; 3) normalizing the input image;

[0124] The image feature encoder includes an image global feature encoding branch, a target feature encoding branch, an inter-target interaction feature encoding branch, an attention mechanism and a feature fusion layer; the image global feature encoding branch includes a plurality of convolution modules, the input of the image global feature encoding branch is a normalized image of the input image, which is used to extract the global features of the image and convert it into a vector form; the target feature encoding branch includes a plurality of convolution modules, the input of the target feature encoding branch is a normalized image of the region where various types of targets are located, which is used to extract local target features and convert it into a vector form; the inter-target interaction feature encoding branch includes a plurality of convolution modules, the input of the inter-target interaction feature encoding branch is a normalized image of the region where various types of targets interact, which is used to extract the features of the inter-target action interaction region and convert it into a vector form; the attention mechanism is used to capture the target features and inter-target interaction features that need to be focused on relative to the global features; the feature fusion layer is used to normalize the above-mentioned image global features, the target features that need to be focused on and the inter-target interaction features that need to be focused on, and then splice them through a fully connected layer to output an image feature vector;

[0125] The feature-text decoder inputs an image feature vector obtained by processing an image feature encoder; and uses a combination module including at least 2 layers of long short-term memory (LSTM) networks to decode the image feature vector into text; the feature-text decoder refers to a decoder from image features to text.

[0126] This embodiment constructs an image factual description model based on the attention mechanism, such as Figure 5 As shown, the specific implementation is as follows:

[0127] (1) The image factual description preprocessing module first preprocesses the input image; then uses a pre-trained target detection algorithm to detect the areas where various targets appearing in the image are located; further, uses a pre-trained target relationship detection algorithm to detect the areas where various targets appearing in the image interact; finally, the images of the target area and the interactive relationship area between the targets are cropped and aligned, and normalized into an image of 56×56 pixels; the specific implementation is as follows:

[0128] First, the input image is preprocessed and normalized to a 224×224 pixel image;

[0129] Next, we use the Faster-RCNN pre-trained on the COCO dataset as the target detector to extract the target region F in the image. O , and determine the category of the target contained in the image;

[0130] Afterwards, we use the Faster-RCNN pre-trained on the Open Images dataset as the backbone network and adopt a pre-trained network with two fully connected heads + SoftNMS to extract the region F where the objects in the image interact. R ;

[0131] Finally, the images of the target area and the interaction area between targets are cropped and aligned, and normalized to 56×56 pixel images;

[0132] The image feature encoder includes an image global feature encoding branch, a target feature encoding branch, an inter-target interaction feature encoding branch, an attention mechanism, and a feature fusion layer;

[0133] (2) The image global feature encoding branch includes a plurality of convolution modules, the input of which is a normalized image of the input image, and is used to extract the global features of the image and convert them into a vector form;

[0134] Convolution module d9: includes 2 convolution layers and 1 pooling layer. Both convolution layers use 64 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 224×224×64. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 112×112×64.

[0135] Convolution module d10: includes 2 convolution layers and 1 pooling layer. Both convolution layers use 128 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping. The output size is 112×112×128 feature maps. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2. The output size is 56×56×128 feature maps.

[0136] Attention module 1: Use the spatial attention mechanism to process the 14×14×256 global features of the image output by the convolution module d10. The specific implementation is as follows:

[0137] First, two 14×14×1 feature layers are obtained through global maximum pooling and global average pooling. Then, the two feature layers are stacked, and the number of channels of the stacked 14×14×2 feature layer is adjusted by 1×1 convolution to obtain a 14×14×1 feature layer. Finally, a 14×14×1 global spatial attention mechanism is output through sigmoid, and the obtained global spatial attention mechanism is multiplied with the original input feature to obtain the final image global feature map based on the spatial attention mechanism, with a size of 14×14×256.

[0138] Convolution module d11: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 256 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 56×56×256. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 28×28×256.

[0139] Convolution module d12: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 512 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps with a size of 28×28×512. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps with a size of 14×14×512.

[0140] Convolution module d13: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 512 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps with a size of 14×14×512. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps with a size of 7×7×512.

[0141] (3) The target interaction feature encoding branch includes multiple convolution modules and an attention module. The input of this branch is a normalized image of the region where each type of target interaction is located. It is used to extract the target interaction features and convert them into vector form. Specifically,

[0142] Convolution module d14: includes 2 convolution layers and 1 pooling layer. Both convolution layers use 128 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 56×56×128. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 28×28×128.

[0143] Convolution module d15: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 256 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 28×28×256. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 14×14×256.

[0144] Attention module 2: According to the global features based on the spatial attention mechanism output by attention module 1, the non-important target interaction feature maps are removed, and the channel attention mechanism is used to focus on the target interaction feature map with a size of 14×14×256 output by convolution module d15. The specific implementation is as follows:

[0145] First, two 1×1×256 feature layers are obtained through global average pooling; then, the above feature layers are fully connected twice. The number of channels of the first full connection is small, about 150, and the number of channels of the second full connection is 256, and finally a 1×1×256 feature layer is output; finally, a 1×1×256 target interaction relationship channel attention mechanism is output through sigmoid, and the channel attention mechanism of the target interaction relationship is multiplied by the original input feature to obtain the final target interaction feature map based on the channel attention mechanism, with a size of 14×14×256;

[0146] Convolution module d16: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 512 3×3 convolution kernels to perform convolution operations on the interaction feature map of the focused targets. The convolution step is 1, the zero padding edge length is 1, and the convolution is performed after ReLU nonlinear mapping. The output size is 14×14×512 feature map; the pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature map with a step size of 2, and outputs a 7×7×512 feature map of the interaction feature map of the focused targets.

[0147] (4) The target feature encoding branch includes multiple convolution modules. The input of this branch is a normalized image of the area where each type of target is located. It is used to extract local target features and convert them into vector form. The specific implementation is as follows:

[0148] Convolution module d17: includes 2 convolution layers and 1 pooling layer. Both convolution layers use 128 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding edge length is 1, and the convolution is performed after ReLU nonlinear mapping. The output size is 56×56×128 feature map; the pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature map with a step size of 2, and the output size is 28×28×128 feature map;

[0149] Convolution module d18: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 256 3×3 convolution kernels to perform convolution operations on feature maps. The convolution step is 1, the zero padding and edge length is 1, and the convolution is performed through ReLU nonlinear mapping to output feature maps of size 28×28×256. The pooling layer uses a 2×2 maximum pooling kernel and downsamples the feature maps with a step size of 2 to output feature maps of size 14×14×256.

[0150] Attention module 3: According to the global features based on the spatial attention mechanism output by attention module 1, the unimportant target feature maps are removed, and the channel attention mechanism is used to process the focused target feature maps of size 14×14×256 output by convolution module d18. The specific implementation is as follows:

[0151] First, two 1×1×256 feature layers are obtained through global average pooling; then, the above feature layers are fully connected twice. The number of channels of the first full connection is small, about 150, and the number of channels of the second full connection is 256, and finally a 1×1×256 feature layer is output; finally, a 1×1×256 target channel attention mechanism is output through sigmoid, and the obtained target channel attention mechanism is multiplied with the original input feature to obtain the final target-to-target feature map based on the channel attention mechanism, with a size of 14×14×256;

[0152] Convolution module d19: includes 3 convolution layers and 1 pooling layer. All 3 convolution layers use 512 3×3 convolution kernels to perform convolution operations on the key target feature map. The convolution step is 1, the zero padding edge length is 1, and the convolution is performed after ReLU nonlinear mapping. The output size is 14×14×512 feature map; the pooling layer uses a 2×2 maximum pooling kernel, and downsamples the feature map with a step size of 2, and outputs a key target feature map with a size of 7×7×512.

[0153] (5) The feature fusion module includes multiple feature fusion layers and pooling layers, which are specifically implemented as follows:

[0154] The feature fusion layer c3 takes as input the focused target interaction feature map output by the target interaction feature encoding branch and the image global feature output by the image global feature encoding branch, both of which have a size of 7×7×512. The two feature maps are respectively subjected to average pooling operations, each of which has a size of 4×4×512. The pooled image global feature and target interaction feature map are added to obtain a feature map of size 4×4×512 after feature fusion; and the focused target interaction feature map of size 4×4×512 is output at the same time.

[0155] The feature fusion layer c4 takes as input the focused target feature map output by the target feature encoding branch and the image global feature output by the image global feature encoding branch, both of which have a size of 7×7×512. The two feature maps are respectively subjected to average pooling operations, each of which has a size of 4×4×512. The pooled image global feature and the focused target feature map are added together to obtain a feature map of size 4×4×512 after feature fusion; and the focused target feature map of size 4×4×512 after pooling is output at the same time.

[0156] The upsampling layer e1 and the feature fusion layer c3 output a 4×4×512 feature map after fusion, which is upsampled to a 7×7×512 feature map.

[0157] The upsampling layer e2 and the feature fusion layer c4 output a 4×4×512 feature map after fusion, which is upsampled to a 7×7×512 feature map.

[0158] The feature fusion layer c5 first inputs the feature map of size 7×7×512 output by the upsampling layer e1, the feature map of size 7×7×512 output by the convolution module d13, and the feature map of interaction between focused targets after pooling of size 4×4×512 output by the feature fusion layer c3; then, the two input feature maps of size 7×7×512 are stacked to obtain a new global feature map of size 7×7×512; finally, a global average pooling operation is performed on the global feature map and the interaction feature map between focused targets, respectively, to obtain two 512-dimensional feature vectors, and the two feature vectors are concatenated to output a 1024-dimensional feature vector;

[0159] The feature fusion layer c6 first inputs the feature map of size 7×7×512 output by the upsampling layer e2, the feature map of size 7×7×512 output by the convolution module d13, and the feature map of the focused target after pooling of size 4×4×512 output by the feature fusion layer c4; then, the two input feature maps of size 7×7×512 are stacked to obtain a new global feature map of size 7×7×512; finally, a global average pooling operation is performed on the global feature map and the focused target feature map respectively to obtain two 512-dimensional feature vectors, and the two feature vectors are concatenated to output a 1024-dimensional feature vector;

[0160] The feature fusion layer c7 concatenates the two 1024-dimensional vectors output by the feature fusion layer c5 and the feature fusion layer c6 to obtain a 2048-dimensional feature vector;

[0161] The feature-text decoder inputs the image feature vector obtained by processing the image feature encoder; and uses a combination module including at least two layers of long short-term memory (LSTM) networks to decode the image feature vector into text; the specific algorithm flow is as follows:

[0162] Given the feature F according to the attention mechanism, the decoder output is expressed as follows:

[0163]

[0164] E(·) is the word embedding function, is the total state of all output results, and the calculation method of its attention weight can be set as:

[0165]

[0166] W attention =softmax(score(F,h t F ))

[0167] Then the feature attention A(t) at the specific location generation step t can be expressed as the following formula:

[0168]

[0169] On this basis, the target features, target interaction features and global features based on the attention mechanism can be expressed as Based on the attention mechanism, the interactive relationship characteristics between The εth relation feature in It can be expressed as the following formula, where W is the weight matrix;

[0170]

[0171] The final output is a factual description of "a woman standing in the garden holding a baby";

[0172] Step 5: Use the image description dataset Ai-Challenger Caption to train the image factual description model;

[0173] This embodiment uses the Ai-Challenger Caption image description dataset. The Ai-Challenger Caption image description dataset has a five-sentence Chinese description for each given image. The dataset contains 300,000 images and 1.5 million Chinese descriptions. The training set contains 210,000 images, the validation set contains 30,000 images, the test set A contains 30,000 images, and the test set B contains 30,000 images; in practice, other image description datasets can also be used, or image description datasets can be collected by yourself to establish an image description dataset with Chinese description labels.

[0174] Step 6: construct an emotional image description initialization module. This module selects emotional words corresponding to the image emotion categories from the emotional word embedding library according to the image emotion categories output by the trained image emotion recognition model, and embeds them into the image factual description output by the trained image factual description model to generate an initial emotional image description.

[0175] In this embodiment, the text sentiment detector AYLIEN API is used to perform sentiment detection on the sentence S generated in step 4, and the one-hot vector J is used. T represents the position k of the sentiment word, and the features of the object it modifies and the interaction features between the corresponding objects are J T The vector dimension is the length L of S;

[0176] Construct the basis vector J of 8 categories of emotions S , and simultaneously detect the similarity between the emotion words and the emotion basis vectors. k is the number of emotion words to be detected in S, and the calculation formula is:

[0177]

[0178] use The emotion basis vector with the closest similarity is extracted to compare whether it is the same as the image emotion output by the image emotion recognition model.

[0179] If they are the same, it is used as the initial emotional description result by default; if they are not the same, the emotional vocabulary corresponding to the image detection emotion is searched for from the emotional vocabulary S-corpus according to the association mapping relationship between verbs and nouns, and the replacement result is used as the initial emotional description sentence;

[0180] If the result generated in step 4 fails to detect emotional words, the emotional words corresponding to the detected emotions in the image are directly searched from the emotional word library S-corpus according to the associative mapping relationship between verbs and nouns and added to the corresponding relationship area. Finally, the length of the initial emotional image description sentence X generated is L′, and the initial emotional image description sentence is "a tranquil woman holding a baby standing in a leisurely garden".

[0181] Step 7: Build Figure 6 The reinforcement learning-based fine-tuning module shown includes a sentence reconstruction generator, a sentence storage unit, a sentence sampling unit, a sentence evaluation unit, and a word selection evaluation unit; wherein the sentence reconstruction generator serves as an agent in the reinforcement learning system, and the sentence storage unit, the sentence sampling unit, the sentence evaluation unit, and the word selection evaluation unit constitute the external environment in the reinforcement learning system; the sentence reconstruction generator continuously interacts with the external environment, obtains the reward information of the external environment, learns the mapping from the environment state to the behavior action, optimizes and adjusts the behavior action, adjusts the initial emotional image description generated in step six, and generates the final emotional image description. The specific steps are as follows:

[0182] 1) The statement reconstruction generator is based on the environment state (State) S at time t-1 t-1 , that is, the statement S generated at time t-1 t-1 , and the reward (Reward) R at the t-1th moment t-1 , that is, the score R of the word selected at time t-1 t-1 , select words with similar semantics from the sentiment word embedding library through the word selector, execute the "action" of word selection, and add the selected words to the sentence S generated at time t-1 t-1 In the example above, the statement S at time t is generated. t ; Among them, the sentence S0 generated at time 0 is the start symbol of sentence generation; the word selector "Action" selects words with similar semantics based on the known semantics of the target sentence and the recorded sentence of the previous moment and its evaluation results, where the degree of semantic similarity is represented by the distance between semantic word vectors; the word selected at time t is a t It means that at time t, based on the generated t-1 words, y t As the operation of the tth word to be generated; word y t Taken from the target vocabulary C + =(C-corpus)∪(S-corpus). The updated statement s at time t as state feedback tIndicates the action (Action) t After implementation, y t Add the statement s at time t-1 t-1 The newly generated sentence at time t; the selected word score as reward (Reward) is the updated sentence s at time t of state (State) feedback t Indicates the action (Action) t After implementation, y t Add the statement s at time t-1 t-1 The newly generated sentence at time t; specifically, each word y t Corresponding to state S t The score is calculated by the word selection evaluation unit in the external environment.

[0183] In this embodiment, construct Figure 6 The sentence reconstruction generator as an agent is used to reconstruct the input initial emotional image description sentence; the sentence reconstruction generator network structure adopts a double-layer recurrent neural network combined with an attention mechanism; a deterministic strategy is adopted, P θ (y t |S t ) indicates state S t Given word y t ; L" is the total length of the sentence, μ is the function that reduces the reward as the sentence length decreases, then the total reward during the sentence reconstruction generator training process is It can be expressed as:

[0184]

[0185] The gradient during the optimization process is θ is the parameter of the generator;

[0186] 2) The statement storage unit stores the updated statement S at time t t The sentence sampling unit updates the sentence S at the tth moment based on the sampling search algorithm. tRolling Out is performed to generate N sentences, where N is 3, 4 or 5; the sampling search algorithm of the sentence sampling unit can adopt polynomial sampling or Monte Carlo sampling method; the sentence evaluation unit first evaluates and scores the N sentences generated by the sentence sampling unit using the emotion identifier, grammatical collocation identifier and semantic identifier respectively, and obtains N emotion reward scores, grammatical collocation reward scores and semantic reward scores, and then adopts the weighted average method to obtain the comprehensive reward score, and finally inputs the comprehensive reward score into the word selection evaluation unit; the word selection evaluation unit outputs the score of the selected word as the reward (Reward) R fed back by the external environment (Environment) to the sentence reconstruction generator t ;

[0187] In this embodiment, a Monte Carlo-based random beam search method is used to generate sentences, and the number of generated sentences is N. sampling If sampling starts at time t, it is expressed as N sampling The number of can be set to 3-5, and N is used in the embodiment of the present invention. sampling 3; the statement evaluation unit is used to evaluate the generated N sampling Complete sampled sentences are evaluated and scored to obtain the emotional reward scores, semantic reward scores and grammatical collocation reward scores of the N sampled sentences, and then the weighted average method is used to obtain the comprehensive reward score; finally, a reward basis is provided for the evaluation of word selection in the sentence reconstruction unit; the semantic discriminator D1 uses the word shift distance WMD calculation, and the specific formula is as follows:

[0188]

[0189] L″ represents the target emotional image description sentence Y={y1,…,y L″ The emotion discriminator D2 is trained on the sentiment140 dataset using an adversarial neural network to identify the emotion category of the generated sentence. The loss function during the training process can be set as follows:

[0190]

[0191] is the generator result, To mark the true value, the reward evaluation module can obtain the sentence emotion detection, semantic reward results and grammatical collocation reward results: D1(Y), D2(Y) and D3(Y);

[0192] The grammatical collocation identifier is composed of a double-layer recurrent neural network pre-trained by the grammatical collocation corpus CCL (Centre for Chinese Linguistics);

[0193] If the generated t-1 target sequence vocabulary states are recorded as S t =(X,Y 1:t-1 ), then we can get the state S t The behavior y of choosing a word at time t t The reward calculation formula is:

[0194]

[0195] Since both emotions and semantics are very important, α and β can be set to values ​​greater than 0.5, and η can be set to a value between 0.2 and 0.5.

[0196] The statement storage unit is used to store updated statements, and the storage unit size is L";

[0197] The word selection evaluation unit uses the sentence evaluation score f(S) output by the sentence evaluation unit at the current time t t ,y t ) minus the sentence evaluation score f(S) output by the sentence evaluation unit at the previous time t-1 t-1 ,y t-1 ), and get the score of the word selected at the current moment γ(S t ,y t ), specifically expressed as:

[0198] γ(S t ,y t )=f(S t ,y t )-f(S t-1 ,y t-1 )

[0199] The update statement reconstructs the total reward during the training of the generator G for:

[0200]

[0201] The gradient update during the optimization process is:

[0202]

[0203] 3) Iterate step 1) to step 2), the sentence reconstruction generator continuously interacts with the external environment until the maximum reward for sentence reconstruction is obtained, and the final emotional image description is generated.

[0204] A fine-tuning module based on reinforcement learning constructed in this embodiment, when t=1 and t=2, the operation process of the fine-tuning module based on reinforcement learning is as follows:

[0205] The sentence reconstruction generator starts at time t=1. First, based on the initial emotional image description "a tranquil woman holding a baby standing in a leisurely garden" output by the emotional image description initialization module, the sentence state and word selection are initialized. Suppose the word selector at time t=1 selects candidate words similar to the first word "one" in the initial emotional image description from the vocabulary, such as "one", "single", etc., and iteratively evaluates each candidate word. Taking "one" as an example, suppose the word selector selects "one" at this time, and the word selection is initially set to the first word a1←y1="one", and the sentence S1 at time t=1 is "one"; then, for the sentence S1 "one", the Monte Carlo random beam search is used. Sentence generation is performed in this way, and three complete sentences based on the sentence state S1 "one" are generated: (1) A beautiful woman is in the garden...; (2) A bored woman is in the flower field...; (3) A beautiful woman is in the flower field...; Then, the sentence evaluation unit is used to score the above three sentences from the perspectives of sentence emotion, sentence semantics and sentence grammatical collocation, and the sentence rewards f(S1, y1) are 0.8, 0.2 and 0.8 respectively; the average value is used to obtain the comprehensive score of f(S1, y1) of 0.6; finally, the result is fed back to the word selection evaluation unit in the sentence reconstruction unit and the comprehensive score of the current sentence and the sentence reward f(S1, y1) is recorded; when t = 2, first, extract the recorded The emotional image description initialization module outputs the initial emotional image description "a tranquil woman holding a baby standing in a leisurely garden" and sentence S1 "one". Afterwards, the word selector selects a word from similar emotional words such as "beautiful" and "beautiful" as the word selector behavior at time t = 2 based on the result of f(S1, y1) and the words generated by the rollback of sentence S1. Assume that the current selection is a2←y2="beautiful". Then, "beautiful" is combined with the sentence to generate the updated sentence S2 "a beautiful". The Monte Carlo random beam search method is used to generate three complete sentences based on sentence S2 "a beautiful": (1) A beautiful woman in the garden...; (2) A beautiful The lady is in the flower field...; (3) A beautiful woman is in the flower field...; Finally, the sentence evaluation unit is used to score the above three sentences from the perspectives of sentence emotion, sentence semantics and sentence grammatical collocation, and the sentence rewards f(S2, y2) are 0.9, 0.9 and 0.9 respectively; the average value is used to obtain the comprehensive score of f(S1, y1) of 0.9; the result is fed back to the word selection evaluation unit and the current sentence state and the comprehensive score of the sentence reward f(S2, y2) of 0.9 are recorded; the score of word y2 corresponding to state S2 is γ(S2, y2) = f(S2, y2) - f(S1, y1) = 0.3, which means that beauty has a positive effect on the sentence state; at the same time, the sentence reconstruction generator uses the formula Update the behavior reward; and continue to select words from the emotional word embedding library until the objective function value of the total reward is maximized. Finally, the final emotional image description "a beautiful woman holding a baby standing in a blooming garden" is generated through reinforcement learning fine-tuning.

[0206] Based on the same inventive concept, the structure diagram of the emotional image description system based on reinforcement learning disclosed in the embodiment of the present invention is as follows: Figure 2 As shown, it includes at least one computing device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the above-mentioned emotional image description method based on reinforcement learning is implemented.

[0207] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An emotional image description method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Build an emotional word embedding library based on a large-scale corpus; Step 2: Build an image emotion recognition model; Step 3: Use the image emotion analysis dataset to train the image emotion recognition model; Step 4: construct an image factual description model based on an attention mechanism for generating image factual descriptions, wherein the image factual description model includes an image factual description preprocessing module, an image feature encoder, and a feature-text decoder which are sequentially connected; Step 5: Use the image description dataset to train the image factual description model; Step 6: construct an emotional image description initialization module. The emotional image description initialization module selects emotional words corresponding to the image emotion category from the emotional word embedding library according to the image emotion category output by the trained image emotion recognition model, and embeds them into the image factual description output by the trained image factual description model to generate an initial emotional image description; Step 7: Construct a fine-tuning module based on reinforcement learning, which is used to fine-tune the initial emotional image description to generate the final emotional image description; In step 7, the fine-tuning module based on reinforcement learning includes a sentence reconstruction generator, a sentence storage unit, a sentence sampling unit, a sentence evaluation unit, and a word selection evaluation unit. The specific method for the fine-tuning module to fine-tune the initial emotional image description is as follows: Step 701: The sentence reconstruction generator selects words with similar semantics from the sentiment word embedding library through the word selector according to the environmental state at time t-1 and the reward at time t-1, performs the word selection action, and adds the selected words to the sentence S generated at time t-1. t-1 In the example above, the statement S at time t is generated. t ; Among them, the statement S0 generated at time 0 is the statement generation start symbol, and the environment state at time t-1 is the statement S generated at time t-1. t-1 , the reward R at time t-1 t-1 That is, the score of the word selected at time t-1, where t is the time; Step 702: The statement storage unit stores the updated statement S at time t. t The sentence sampling unit updates the sentence S at the tth moment based on the sampling search algorithm. t Roll back to generate N sentences, where N is 3, 4 or 5; the sentence evaluation unit first evaluates and scores the N sentences generated by the sentence sampling unit using the emotion identifier, grammatical collocation identifier and semantic identifier to obtain N emotion reward scores, grammatical collocation reward scores and semantic reward scores, and then takes the weighted average method to obtain the comprehensive reward score, and finally inputs the comprehensive reward score into the word selection evaluation unit; the word selection evaluation unit outputs the score of the selected word as the reward R fed back by the external environment to the sentence reconstruction generator t ; Step 703, iterate step 701 to step 702, the sentence reconstruction generator continuously interacts with the external environment until the maximum reward for sentence reconstruction is obtained, and the final emotional image description is generated.

2. The emotional image description method based on reinforcement learning according to claim 1, characterized in that: In step seven, the fine-tuning module based on reinforcement learning includes a sentence reconstruction generator, a sentence storage unit, a sentence sampling unit, a sentence evaluation unit and a word selection evaluation unit; wherein, the sentence reconstruction generator serves as an intelligent agent in the reinforcement learning system, and the sentence storage unit, the sentence sampling unit, the sentence evaluation unit and the word selection evaluation unit constitute the external environment in the reinforcement learning system; the sentence reconstruction generator continuously interacts with the external environment, obtains the reward information of the external environment, learns the mapping from the environment state to the behavior action, optimizes and adjusts the behavior action, fine-tunes the initial emotional image description, and generates the final emotional image description.

3. The emotional image description method based on reinforcement learning according to claim 1, characterized in that: The sampling search algorithm of the sentence sampling unit adopts polynomial sampling or Monte Carlo sampling method.

4. The emotional image description method based on reinforcement learning according to claim 1, characterized in that: In step 1, the specific method of building the sentiment word embedding library is as follows: Step 101: Use the NLTK tool to obtain nouns and verbs in the target detection and image description datasets, generate a semantic vocabulary, and calculate the word vector of each semantic word therein; Step 102: Filter out emotion words from the large-scale corpus LSCC, generate an emotion word library, and calculate the emotion word vector of each emotion word; classify the emotion words corresponding to each semantic word in the semantic word library into 8 categories defined by IAPS: pleasure, rage, surprise, acceptance, hatred, ecstasy, fear, and grief; Step 103: Filter out emotion phrases of different emotion categories corresponding to semantic words from the emotion word library, and construct an emotion word embedding library.

5. The emotional image description method based on reinforcement learning according to claim 1, characterized in that: The image emotion recognition model includes an image emotion recognition preprocessing module, a facial emotion feature extraction module, an image theme color feature extraction module, an image emotion feature extraction module, a feature fusion layer, a fully connected layer, and a classification layer; the image emotion recognition preprocessing module includes a face detection unit, a face image normalization processing unit, and an image size normalization processing unit; wherein, The face detection unit detects the face area in the input image using a pre-trained face detection network and labels different face areas; The face image normalization processing unit is used to crop, align and normalize the size of each detected face area; The image size normalization processing unit is used to normalize the size of the input image; The facial emotion feature extraction module is used to extract facial emotion features of each person in the cropped, aligned and size-normalized facial image; The image theme color feature extraction module is used to extract the theme color features of the input image; The image emotion feature extraction module is used to extract the emotion feature of the size-normalized image output by the image size normalization processing unit; The feature fusion layer is used to fuse the facial emotion features output by the face emotion feature extraction module, the theme color features output by the image theme color feature extraction module, and the emotion features output by the image emotion feature extraction module to obtain a fused emotion feature vector; The fully connected layer is used to fully connect the feature fusion layer and the classification layer; The classification layer is used to output the emotion category to which the image belongs.

6. The emotional image description method based on reinforcement learning according to claim 5, characterized in that: In step 2, the specific method of using the image theme color feature extraction module to extract the theme color features of the input image is as follows: Step 1: Use the microelement method to cut the RGB space into independent three-dimensional blocks; Step 2: Put the RGB format pixel scattered points of the image into the cut RGB space, and use the scattered point value as the value of the three-dimensional block. If there is no scattered point in the three-dimensional block, use the center value of the three-dimensional block area as the value of the three-dimensional block. Step 3: weighted summing the values ​​of the three-dimensional blocks in the entire sliding window area is performed by weighted sliding window method to obtain the value of the three-dimensional block of the sliding window size. The size of the sliding window depends on the number of types of image theme colors to be finally selected. Step 4: Through steps 1 to 3, the image theme color features of the input image are finally obtained.

7. The emotional image description method based on reinforcement learning according to claim 6, characterized in that: The RGB space is cut using the microelement method to form independent three-dimensional blocks, which are pixel-sized cubes.

8. The emotional image description method based on reinforcement learning according to claim 1, characterized in that: The specific method of constructing an image factual description model in step 4 is as follows: Step 4.1, the image factual description preprocessing module uses the network model pre-trained on the target detection and target relationship detection data sets to pre-process the input image; the specific method is as follows: 1) using the pre-trained target detection algorithm to detect the areas where various targets appearing in the image are located; using the pre-trained target relationship detection algorithm to detect the areas where various targets appearing in the image interact; 2) cropping and aligning the input image, the images of the areas where the various types of targets are located, and the images of the areas where the various types of targets interact are located, and performing normalization processing to obtain the normalized images of the input image, the normalized images of the areas where the various types of targets are located, and the normalized images of the areas where the various types of targets interact are located; Step 4.2, construct an image feature encoder, which includes an image global feature encoding branch, a target feature encoding branch, an inter-target interaction feature encoding branch, an attention mechanism and a feature fusion layer; the image global feature encoding branch includes a plurality of convolution modules, the input of the image global feature encoding branch is a normalized image of the input image, which is used to extract the global features of the image and convert it into a vector form; the target feature encoding branch includes a plurality of convolution modules, the input of the target feature encoding branch is a normalized image of the area where various types of targets are located, which is used to extract local target features and convert them into vector form; the inter-target interaction feature encoding branch includes a plurality of convolution modules, the inter-target interaction feature encoding branch The input of the branch is a normalized image of the area where various types of target interactions are located, which is used to extract the features of the action interaction area between targets and convert them into vector form; the convolution module includes one or more convolution layers and a pooling layer; the attention mechanism is used to capture the target features and target interaction features that need to be focused on relative to the global features; the feature fusion layer is used to normalize the above-mentioned image global features, the focused target features and the focused target interaction features, respectively, and then splice them through the fully connected layer to output an image feature vector; the fully connected layer fully connects the output of the feature fusion layer to the c output neurons of this layer, and outputs a c-dimensional feature vector; Step 4.3, construct a feature-text decoder, wherein the input of the feature-text decoder is the image feature vector obtained by processing the image feature encoder; and use a combination module including at least 2 layers of long short-term memory (LSTM) networks to decode the image feature vector into text.

9. An emotional image description system based on reinforcement learning, characterized in that: include: Emotional word embedding library, which is built on the basis of a large-scale corpus to provide corpus support for the final generation of emotional image description; The image emotion recognition model comprises an image emotion recognition preprocessing module, a face emotion feature extraction module, an image theme color feature extraction module, an image emotion feature extraction module, a feature fusion layer, a fully connected layer and a classification layer; the image emotion recognition preprocessing module comprises a face detection unit, a face image normalization processing unit and an image size normalization processing unit; wherein the face detection unit detects the face area in the input image by using a pre-trained face detection network and labels different face areas; the image size normalization processing unit is used to normalize the input image. The pixel size of the input image is normalized to obtain a unified image input size; the face image normalization processing unit is used to crop, align and normalize the size of each detected face area; the face emotion feature extraction module includes multiple convolution modules; the image theme color feature extraction module uses a color clustering method to extract theme color features from the training set images in the image emotion analysis data set; the fully connected layer is used to fully connect the feature fusion layer and the classification layer; the classification layer is used to output the emotion category to which the image belongs; finally, the image emotion analysis data set is used to train the image emotion recognition model; The image factual description model includes an image factual description preprocessing module, an image feature encoder and a feature-text decoder; the image factual description preprocessing module preprocesses the input image; the image feature encoder includes an image global feature encoding branch, a target feature encoding branch, an inter-target interaction feature encoding branch, an attention mechanism and a feature fusion layer; the image global feature encoding branch is used to extract the global features of the image and convert them into a vector form; the target feature encoding branch is used to extract local target features and convert them into a vector form; the inter-target interaction feature encoding branch is used to extract The action interaction area features are converted into vector form; the attention mechanism is used to capture the target features and target interaction features that need to be focused on relative to the global features; the feature fusion layer is used to normalize the above-mentioned image global features, target features that are focused on, and target interaction features that are focused on, respectively, and then output an image feature vector through splicing through a fully connected layer; the input of the feature-text decoder is the image feature vector obtained by the image feature encoder; and the image feature vector is decoded into text using the combination module of the long short-term memory LSTM network; finally, the image description dataset is used to train the image factual description model; The emotional image description initialization module selects the emotional words corresponding to the image emotion category from the emotional word embedding library according to the image emotion output by the trained image emotion recognition model, and embeds them into the image factual description output by the trained image factual description model to generate the initial emotional image description; A fine-tuning module based on reinforcement learning uses reinforcement learning to adjust the generated initial emotional image description; the reinforcement learning fine-tuning module includes a sentence reconstruction generator, a sentence storage unit, a sentence sampling unit, a sentence evaluation unit and a word selection evaluation unit; wherein the sentence reconstruction generator serves as an intelligent agent in the reinforcement learning system, and the sentence storage unit, the sentence sampling unit, the sentence evaluation unit and the word selection evaluation unit constitute the external environment in the reinforcement learning system; the sentence reconstruction generator continuously interacts with the external environment, obtains the reward information of the external environment, learns the mapping from the environment state to the behavior action, optimizes and adjusts the behavior action, fine-tunes the initial emotional image description, and generates the final emotional image description.