Humorous image description method based on emotion fusion and feature weight guidance

Abstract

The present invention discloses a humorous image description method based on emotion fusion and feature weight guidance. First, multi-layer convolution and pooling operations are used to extract the humorous feature vector of the input image; at the same time, given an emotional image with human emotional information, a convolutional neural network is used to construct an emotional marker vector; the humorous feature vector and the emotional marker vector are subjected to feature fusion to form a joint feature vector, which is mapped into a long short-term memory network after dimensional alignment is achieved through a linear combination transformation. At each time step, the humorous visual information related to a specific input position is captured, and words are generated one by one to finally form a complete image description. For the generated image description, the organizational data of the humorous image description is comprehensively considered, and the joint features of the described image are weighted and strengthened. The model parameters are fine-tuned based on the humor level of the generated image description, thereby improving the model's fine-grained generation of humorous image descriptions.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision technology, and in particular to a humorous image description method based on emotion fusion and feature weight guidance. Background Art

[0002] Image description is a cross-field that combines computer vision and natural language processing in the field of artificial intelligence. It requires computers to accurately and fluently describe the content in images in natural language. With the development of the Internet and communication technology, there is a widespread demand for inferring the meaning of a large number of images. By establishing a relationship between image information and text information, computers can recognize the semantic information of images and generate corresponding natural language text that people can understand. Deep learning is increasingly being used in various fields and has shown better performance than traditional technologies. With the continuous development of deep learning models, image description has been greatly improved in terms of diversity and naturalness. Especially in the description of humorous images, with the help of deep learning models, new features can be automatically extracted from the input data to realize the advanced emotional function unique to humans: funny humor.

[0003] As an extension of image description, focusing on whether the image description is funny requires understanding the image. Understanding images usually requires interpreting the relationship between objects and their locations, the intention of the content, and linking visual information with prior knowledge. Humorous image description requires a high level of cognitive abstraction, context awareness and recognition of incongruity. Humor is an abstract, subjective and context-dependent cognitive structure involving multiple cognitive factors. Therefore, it is associated with human emotions and the fusion of emotional feature vectors creates new opportunities for quantifying and understanding processes in image description.

[0004] The image description feature set constructed by traditional methods cannot fully explore the connection between non-continuous features and humor. Therefore, the variable features of the humorous information in the image will play an important role in improving the description effect. On the one hand, the combination of convolutional neural network and recurrent neural network is a successful model for feature extraction and sequential processing. Convolutional neural network is usually used for image processing, while recurrent neural network is used for text processing. These two parts are integrated and work together. The long short-term memory network cooperates with memory cells through input gate, forget gate and output gate, so that the sequence information can be better transmitted backward. The design of three gates overcomes the gradient vanishing problem in recurrent neural network, can better capture the longer dependency in the sequence, and is suitable for image description sequence information modeling. Using the advantage of long short-term memory network that can remember time series features well over a long period of time, sequence learning and feature training of historical operation data can help improve the description accuracy.

[0005] Since "humor" depends on cultural, visual and linguistic stimuli and basic emotional factors, it remains a complex concept. The novelty of humor should be neither too low nor too high, so that most audiences can understand and appreciate humor. It is difficult to achieve this balance. Therefore, humorous image description, as a multimodal humor generation task, can be a useful task for studying deep learning to handle high-level abstraction and the upper limit of creativity. The main goal should be to arouse people's cognitive perception of humor rather than provide factual information. This means that a humorous sentence may be irrelevant to the content of the image, but it is still a valid example. Most of the existing work is based on statistical data, ignoring the influence of factors on people's judgment of humor; for the humorous image description itself, from the perspective of factors affecting humor, it is necessary to make the unclear humorous factors gradually clear and guide the weight reinforcement of joint feature information.

[0006] Generating and interpreting humor is a challenging task. Currently, there is little exploration of the topic of humorous image descriptions, and the analysis techniques considered are relatively intuitive and heuristic methods. The massive amount of image and description data needs to be used to realize their greater value.

[0007] The problem with the existing technology is that for a given image, the humorous image description lacks human emotional information and has a limited degree of humor. The existing technology has a narrow range of feature selection, and the multimodal data processing method is rough and cannot effectively utilize data; when describing an image, it rarely introduces humorous elements, emotional information, and weight guidance elements at the same time. A slight change in any element may cause a drastic fluctuation in the image description. Summary of the invention

[0008] Purpose of the invention: The purpose of the present invention is to provide a humorous image description method based on emotion fusion and feature weight guidance.

[0009] Technical solution: A humorous image description method based on emotion fusion and feature weight guidance of the present invention comprises the following steps:

[0010] Step 1: Input an image for which humorous description needs to be generated, encode humorous information, and extract humorous feature vectors by performing feature analysis on the input image using multi-layer convolution and pooling operations based on a two-dimensional feature plane set with global characteristics;

[0011] Step 2: Input an emotional image with emotional tendency elements, encode the emotional information, use a convolutional neural network to construct an emotional sign for the emotional image, and obtain an emotional sign vector;

[0012] Step 3: The humor feature vector obtained in step 1 and the emotion signature vector obtained in step 2 are fused by using a feature concatenation method to form a joint feature vector.

[0013] Step 4: Use the joint feature vector as feature input and input it into the linear layer for linear combination transformation to achieve dimension alignment, and obtain the aligned joint feature vector;

[0014] Step 5: inject the aligned joint feature vector into the long short-term memory network to analyze the organizational data of the humorous image description;

[0015] Step 6: For the organizational data of humorous image descriptions, relying on the context vector, the previous hidden state, and the generated description, the humorous visual information related to the specific input position is captured at each time step, and words are generated one by one and finally form a complete image description;

[0016] Step 7: for the complete image description, analyze the humor level based on the organizational data of the humorous image description and extract the humor level analysis;

[0017] Step 8: According to the humor level analysis, the joint feature vectors of the described images are integrated and analyzed, the weights are adjusted, the description diversity of the model is enhanced, and the features after weight-enhanced guidance are obtained;

[0018] Step 9: For the features after weight reinforcement guidance, fine-tune the model parameters based on the humor level of the generated image description to improve the granularity of the model's description of humorous images.

[0019] Furthermore, step 1 is as follows: input the emotional image with emotional tendency elements into an Inception-v3 network with the last Softmax layer deleted, and extract a set of feature vectors to become the humor feature vector The formula is:

[0020]

[0021] in, Indicates the part of the image with emotional information Wei said.

[0022] Furthermore, step 2 is as follows: input the emotional image with emotional tendency elements into two convolutional layers using ReLU activation function, then pass through a maximum pooling layer, and finally pass through two fully connected layers using ReLU activation function to extract the emotional sign vector ;

[0023]

[0024] Furthermore, step 3 is as follows: With emotion logo vector Implement feature concatenation to obtain a joint feature vector;

[0025]

[0026] in Each vector corresponds to a part of the task image and the sentiment information image. Wei said.

[0027] Furthermore, step 4 is specifically as follows: first, the new data generated by the input data and weights of each layer is calculated and passed to the next layer, and then combined with the same weights to generate reconstructed data, mapped back to the input layer, and each layer of the network is trained by continuously reducing the error between the input data and the reconstructed data.

[0028] Furthermore, step 5 is specifically as follows: for the unified joint feature vector, map it into the long short-term memory network, use the obtained joint feature vector as feature input, selectively focus on part of the image by selecting a subset of all feature vectors, learn the projection of the weight vector, learn to calculate the weights of the input components, learn the contribution to memory storage, learn to erase the weights of the storage units, and finally control the weights of the storage content to achieve data organization of the joint feature vector.

[0029] Further, step 6 is specifically as follows: for the organizational data of humorous image descriptions, use represents a simple affine transformation with learned parameters, , , , , , , respectively, are the input, forgetting, memory, output, hidden state, and input modulation state of the long short-term memory network; the formula is:

[0030]

[0031]

[0032]

[0033] Among them, the vector is a context vector that captures the visual information relevant to a specific input location, is an embedding matrix, and denote the embedding dimension and the LSTM network dimension respectively, and Represent the Sigmoid activation function and element multiplication respectively, express VI dimensional affine transformation, y t-1 Indicates the last output word, h t-1Represents the previous hidden state;

[0034] Then use the deep output layer to calculate the probability of the output word through the long short-term memory network state, the context vector and the previous word:

[0035]

[0036] in and are randomly initialized learning parameters, It is the first vector of the word sequence of the previous state in the context vector, and finally generates a pair Description of the sequence of encoded words , .in is the size of the vocabulary, is the length of the description, Indicates proportional.

[0037] Furthermore, step 7 is as follows: for the complete image description , a manual evaluation and scoring method is used to analyze the grouping of humor description-related data, and the humor level of the obtained image-text pairs is judged and calibrated as humor scores; the humor level scores annotated in the form of user votes are positive samples; negative samples are composed of image descriptions generated by COCO dataset training. According to the manual evaluation humor score, positive samples and negative samples are distinguished, and finally the humor confidence of the image-text pair is given; the humor score is positively correlated with the humor level of the generated description. If the humor score is higher than the set threshold, it is classified as a high-quality humor description, and if it is lower than the set threshold, it is classified as a low-quality humor description.

[0038] Furthermore, step 8 is specifically as follows: for high-quality humorous descriptions, firstly initialize the weight parameters of the joint feature vectors of the high-quality humorous descriptions, and the initial weights are all 1; set The value of the visual feature parameter vector of the model at the moment The position variable at time is denoted as ; Directly take the expectation of the context vector, the formula is:

[0039]

[0040] This is equivalent to inputting a weighted context vector into the system to adjust the feature weights and take the predicted probability of the next word generated by the generation process in the current state as the immediate benefit. For the joint eigenvectors, for Moment The weight annotation vector of the joint feature vector.

[0041] Further, step 9 is specifically as follows: using the weight-adjusted data feature set obtained in step 8, iteratively training the model until the model converges, and using the high-quality humorous description obtained at the convergence moment The humorous image description result as the final emotional information fusion and feature weight guidance.

[0042] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0043] (1) The present invention combines several related elements and proposes a humorous image description model with emotional information and joint feature weight reinforcement guidance. The humorous feature vector and the emotional marker vector are fused to form a joint feature vector, the features are analyzed on a two-dimensional plane, and the dimensions are aligned through a linear combination transformation and then mapped into a long short-term memory network. The organizational data of the humorous image description is analyzed, and words are generated one by one to finally form a complete image description. For the generated image description, its humor level is analyzed. Based on the features after weight reinforcement guidance, the model parameters are fine-tuned with the humor level of the generated image description as the benefit, thereby improving the model's fine-grained generation of humorous image descriptions.

[0044] (2) The present invention proposes a deep neural network for humorous image description that combines humorous information and emotional information, and applies weight reinforcement guidance to the combined features. By combining computer vision technology, emotional information features are integrated to improve the diversity of humorous style descriptions; a parallel model is used to perform reasoning on different image data to improve reasoning performance; and the weight of the combined features is guided and adjusted. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of a humorous image description method based on emotional information fusion and feature weight guidance;

[0046] Figure 2 It is a network diagram for extracting emotional sign vectors;

[0047] Figure 3 It is a schematic diagram of weight guidance for joint feature vector. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, this method comprises the following steps:

[0050] Step 1: Input the target task image into an Inception-v3 network with the last Softmax layer deleted, and extract a set of feature vectors to become the humor feature vector The formula is .

[0051] Step 2: Extract the emotion marker vector, such as Figure 2 As shown in Figure 1, the input emotional information image enters two convolutional layers using the ReLU activation function, then passes through a maximum pooling layer, and finally passes through two fully connected layers using the ReLU activation function to extract the emotional sign vector. , .

[0052] Step 3: The humor feature vector obtained in step 1 The emotional symbol vector obtained in step 2 Implement feature concatenation to obtain a joint feature vector ,in Each vector corresponds to a part of the task image and the sentiment information image. Wei said.

[0053] Step 4: The joint feature vector obtained in step 3 , a linear layer is used to align the dimensions and transform them to the input size of the LSTM network.

[0054] Step 5, the aligned joint feature vector obtained in step 4 is mapped into the long short-term memory network. The obtained joint feature vector is used as the feature input, and a subset of all feature vectors is selected to selectively focus on certain parts of the image, learn the projection of the weight vector, learn to calculate the weight of the input component, learn the contribution to memory storage, learn to erase the weight of the storage unit, and finally control the weight of the storage content to achieve data organization of the joint feature vector.

[0055] Step 6: For the organizational data obtained in step 5, use represents a simple affine transformation with learned parameters, , , , , , , respectively, are the input, forgetting, memory, output, hidden state, and input modulation state of the long short-term memory network; the formula is:

[0056]

[0057]

[0058]

[0059] vector is a context vector that captures the visual information relevant to a specific input location. is an embedding matrix, and denote the embedding dimension and the LSTM network dimension respectively, and Represents the Sigmoid activation function and element multiplication respectively. Then use the deep output layer to calculate the probability of the output word through the long short-term memory network state, context vector and previous word:

[0060]

[0061] in and are randomly initialized learning parameters. It is the first vector of the word sequence of the previous state in the context vector, and finally generates a pair Description of the sequence of encoded words :

[0062]

[0063] in is the size of the vocabulary, is the length of the description.

[0064] Step 7, description of step 6 ,like Figure 3 As shown in the figure, a manual evaluation and scoring method is used to analyze the grouping of humor description-related data. The humor level of the image-text pair is judged and calibrated as a humor score. The humor level score annotated in the form of user voting is used as a positive sample; the negative sample is composed of image descriptions generated by training the COCO dataset. According to the manual evaluation humor score, positive samples and negative samples are distinguished, and finally the humor confidence of the image-text pair is given. The humor score is positively correlated with the humor level of the generated description. If the humor score is higher than the set threshold, it is classified as a high-quality humor description. If it is lower than the set threshold, it is classified as a low-quality humor description.

[0065] Step 8: For the high-quality humorous description obtained in step 7, first initialize the weight parameters of the joint feature vector of the high-quality humorous description, and the initial weights are all 1. The value of the visual feature parameter vector of the model at the moment The position variable at time is denoted as ; Directly take the expectation of the context vector;

[0066]

[0067] This is equivalent to inputting a weighted context vector into the system to adjust the feature weights. The predicted probability of the next word generated by the generation process in the current state is taken as the immediate benefit.

[0068] Step 9, using the weighted data feature set obtained in step 8, iteratively train the model until the model converges, and use the high-quality humorous descriptions obtained at the convergence moment The humorous image description result as the final emotional information fusion and feature weight guidance.

[0069] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A humorous image description method based on emotion fusion and feature weight guidance, characterized in that: The steps include: Step 1: Input an image for which humorous description needs to be generated, encode humorous information, and extract humorous feature vectors by performing feature analysis on the input image using multi-layer convolution and pooling operations based on a two-dimensional feature plane set with global characteristics; Step 2: Input an emotional image with emotional tendency elements, encode the emotional information, use a convolutional neural network to construct an emotional sign for the emotional image, and obtain an emotional sign vector; Step 3: The humor feature vector obtained in step 1 and the emotion signature vector obtained in step 2 are fused by using a feature concatenation method to form a joint feature vector. Step 4: Use the joint feature vector as feature input and input it into the linear layer for linear combination transformation to achieve dimension alignment, and obtain the aligned joint feature vector; Step 5: inject the aligned joint feature vector into the long short-term memory network to analyze the organizational data of the humorous image description; Step 6: For the organizational data of humorous image descriptions, relying on the context vector, the previous hidden state, and the generated description, the humorous visual information related to the specific input position is captured at each time step, and words are generated one by one and finally form a complete image description; Step 7: for the complete image description, analyze the humor level based on the organizational data of the humorous image description and extract the humor level analysis; Step 8: According to the humor level analysis, the joint feature vectors of the described images are integrated and analyzed, the weights are adjusted, the description diversity of the model is enhanced, and the features after weight-enhanced guidance are obtained; Step 9: For the features after weight reinforcement guidance, fine-tune the model parameters based on the humor level of the generated image description to improve the granularity of the model's description of humorous images.

2. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 1 is as follows: Input the emotional image with emotional tendency elements into an Inception-v3 network with the last Softmax layer deleted, and extract a set of feature vectors to become the humor feature vector ; The formula is: in, Indicates the part of the image with emotional information Wei said.

3. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1 is characterized in that: Step 2 is as follows: Input the emotional image with emotional tendency elements into two convolutional layers using ReLU activation function, then pass through a maximum pooling layer, and finally pass through two fully connected layers using ReLU activation function to extract the emotional sign vector. ; 。 4. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 3 is as follows: With emotion logo vector Implement feature concatenation to obtain a joint feature vector; in Each vector corresponds to a part of the task image and the sentiment information image. Wei said.

5. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 4 is as follows: first, the new data generated by the input data and weights of each layer is calculated and passed to the next layer, and then combined with the same weights to generate reconstructed data, which is mapped back to the input layer. Each layer of the network is trained by continuously reducing the error between the input data and the reconstructed data.

6. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 5 is as follows: for the unified joint feature vector, map it into the long short-term memory network, use the obtained joint feature vector as feature input, selectively focus on part of the image by selecting a subset of all feature vectors, learn the projection of the weight vector, learn to calculate the weights of the input components, learn the contribution to memory storage, learn to erase the weights of the storage units, and finally control the weight of the storage content to achieve data organization of the joint feature vector.

7. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 6 is as follows: for the organizational data of humorous image descriptions, use represents a simple affine transformation with learned parameters, , , , , , , respectively, are the input, forgetting, memory, output, hidden state, and input modulation state of the long short-term memory network; the formula is: Among them, the vector is a context vector that captures the visual information relevant to a specific input location, is an embedding matrix, and denote the embedding dimension and the LSTM network dimension respectively, and Represent the Sigmoid activation function and element multiplication respectively, express VI dimensional affine transformation, y t-1 Indicates the last output word, h t-1 Represents the previous hidden state; Then use the deep output layer to calculate the probability of the output word through the long short-term memory network state, the context vector and the previous word: in and are randomly initialized learning parameters, It is the first vector of the word sequence of the previous state in the context vector, and finally generates a pair Description of the sequence of encoded words , ,in is the size of the vocabulary, is the length of the description, Indicates proportional.

8. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 7 is as follows: For a complete image description ,The manual evaluation scoring method is used to analyze the grouping of humorous description ,related data, and the humor degree of the obtained image-text pairs is determined and ,calibrated as the humor score; The humor level score annotated in the form of user voting is the positive sample; the negative sample is composed of the image description generated by training the COCO dataset. According to the manual evaluation of the humor score, the positive and negative samples are distinguished, and finally the humor confidence of the image-text pair is given; the humor score is positively correlated with the humor level of the generated description. If the humor score is higher than the set threshold, it is classified as a high-quality humor description, and if it is lower than the set threshold, it is classified as a low-quality humor description.

9. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 8 is as follows: for high-quality humorous descriptions, first initialize the weight parameters of the joint feature vector of the high-quality humorous descriptions, and the initial weights are all 1; set The value of the visual feature parameter vector of the model at the moment The position variable at time is denoted as ; Directly take the expectation of the context vector, the formula is: It is equivalent to inputting a weighted context vector into the system to adjust the feature weights and take the predicted probability of the next word generated by the generation process in the current state as the immediate benefit, where For the joint eigenvectors, for Moment The weight annotation vector of the joint feature vector.

10. The humorous image description method based on emotion fusion and feature weight guidance according to claim 1, characterized in that: Step 9 is as follows: use the weighted data feature set obtained in step 8 to iteratively train the model until the model converges, and then use the high-quality humorous descriptions obtained at the convergence moment as The humorous image description result as the final emotional information fusion and feature weight guidance.

Images