A method and device for emotional image retrieval based on image style coding

Abstract

The present invention discloses a method and device for emotional image retrieval based on image style coding, including: obtaining the style semantic features of an image by using the correlation between feature graphs; paying attention to the feature areas related to the emotional distribution in the image by using a multi-order attention mechanism to obtain the content semantic features of the image; fusing the style semantic features with the content semantic features to obtain the emotional features of the image, and obtaining the first emotional distribution prediction result by using the emotional feature coding; transmitting the emotional features as the input of the graph node to the graph neural network, enhancing the emotional features by using the correlation between the emotional tags, and obtaining the second emotional distribution prediction result; fusing the first emotional distribution prediction result and the second emotional distribution prediction result into the final emotional distribution prediction result; using a natural language processing algorithm to calculate the emotional polarity of the input search term, and outputting an emotional image with a high emotional distribution similarity according to the emotional polarity of the input term. The present invention can more accurately predict the emotional distribution of an image and improve the accuracy of emotional image retrieval.

Description

Technical Field

[0001] The present invention relates to the field of multimedia emotion computing, and in particular to an emotion image retrieval method and device based on image style coding. Background Art

[0002] With the advent of the information age, more and more users tend to use images to express their opinions and emotions on social media. However, current image retrieval focuses more on the content of the image, which makes it difficult to retrieve images that match one's emotions on the Internet. Predicting the distribution of emotions reflected in images is of great significance to improving the retrieval accuracy of emotional images. As one of the important research directions in the high-level semantic cognition of computer vision, visual emotion computing is still in its infancy. Its main challenges lie in the huge semantic gap between low-level image features and high-level emotional semantics, as well as the subjectivity and ambiguity of emotions.

[0003] In response to the first challenge, most current algorithms focus on the design of visual feature extractors, using labeled emotional feature regions or attention mechanisms to extract emotion-related regions in images; in response to the second challenge, distributed learning with label relevance has become a new learning paradigm. Label distribution learning is a broader learning paradigm that includes the currently common single-label learning and multi-label learning, and can more effectively express the importance of labels. Using the label distribution learning paradigm to map an image to an emotional label distribution can better reflect the emotional distribution expressed by the image. Nevertheless, how to effectively bridge the semantic gap between low-level visual features and high-level emotional semantics and how to make full use of the correlation between emotional labels are still valuable research directions.

[0004] Therefore, it is meaningful to propose a visual emotion distribution prediction algorithm that can effectively extract image emotion features and fully extract the correlation between emotion labels. Summary of the invention

[0005] The present invention provides an emotional image retrieval method and device based on image style coding. In view of the problem of poor accuracy in retrieval of emotional words in current image retrieval, the present invention introduces an image emotion distribution prediction algorithm, makes full use of image style information as a supplement to image content information, obtains more complete image emotion information, thereby more accurately predicting image emotion distribution and improving the accuracy of emotional image retrieval. The details are described below:

[0006] A method for emotional image retrieval based on image style coding, the method comprising the following steps:

[0007] Based on multiple low-dimensional feature maps of the multi-level feature map, the correlation is calculated to obtain multiple GRAM matrices, and the GRAM matrices are concatenated and passed through the convolutional neural network for style extraction to obtain the style semantic features of the image;

[0008] Use a multi-order attention mechanism to focus on the feature areas in the image that are related to the emotional distribution, and obtain the content semantic features of the image;

[0009] The style semantic features and the content semantic features are integrated to obtain the emotional features of the image, and the emotional features are encoded to obtain the first emotional distribution prediction result;

[0010] The sentiment features are transmitted to the graph neural network as the input of the graph node, and the sentiment features are enhanced by using the correlation between the sentiment labels to obtain the second sentiment distribution prediction result;

[0011] The first sentiment distribution prediction result and the second sentiment distribution prediction result are merged into a final sentiment distribution prediction result;

[0012] A natural language processing algorithm is used to calculate the sentiment polarity of the input search terms, and an emotional image with a high degree of sentiment distribution similarity is output based on the sentiment polarity of the input terms.

[0013] Wherein, the multi-level feature map is:

[0014] The visual emotion distribution dataset is divided into training set and test set, and the multi-level feature map of the input image is extracted through the feature extraction network.

[0015] Furthermore, the GRAM matrix is:

[0016] F style =f style (Concat(G 0 ,G 1 ,G 2 ))

[0017] Among them, G k , k = 0, 1, 2 represents the GRAM style matrix calculated from the feature map output by the kth convolutional block; Concat(·) represents the concatenation of multiple levels of GRAM style matrices in the channel dimension; f style represents the image style encoding network, which consists of two convolutional blocks, each of which contains a convolutional layer, an instance normalization layer, and a pooling layer; F style Represents the final generated image style information encoding.

[0018] The multi-order attention mechanism is used to focus on the feature areas in the image that are related to the emotional distribution, and the content semantic features of the image are obtained as follows:

[0019]

[0020] Where A(x) represents the attention score at position x in the image, sigmoid(·) represents the activation function, is a learnable parameter, in T represents transpose, r represents the order, R is the highest order, c is the number of channels, d is the dth first-order tensor, is a real number space, z is the result of the action of different order descriptors, s is the order variable, u s is the decomposed vector and x is the local descriptor.

[0021] The style semantic features and the content semantic features are fused to obtain the emotional features of the image, and the first emotional distribution prediction result is obtained by encoding the emotional features:

[0022]

[0023] in, Represents the r-th order image content coding information; f ss (·) used to map the concatenated image style encoding and image content encoding into category-specific image emotion encoding; represents the category-specific image emotion encoding, F style Represents the style encoding of the image, and Concat means concatenating the two features in the channel dimension.

[0024] Furthermore, the graph neural network is composed of a static graph network and a dynamic graph network cascaded together.

[0025] The static graph network is as follows:

[0026]

[0027] Among them, A s is the adjacency matrix extracted from the training set, W s represents the weight matrix of the static graph network, f SGCN Represents a static graph network;

[0028] The dynamic graph network is as follows:

[0029]

[0030] Among them, A d It is composed of static graph network H r The adjacency matrix of the generated dynamic graph, A d =f d (H r ), W d is the weight matrix of the dynamic graph network, fd Generate a network for the dynamic adjacency matrix, H r is the node feature matrix output by the static graph network, f DGCN A dynamic graph network.

[0031] An emotional image retrieval device based on image style coding, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the method steps described in the first aspect.

[0032] The beneficial effects of the technical solution provided by the present invention are:

[0033] 1. Use the GRAM matrix to calculate the correlation between feature maps at each level, and use the correlation between feature maps across levels of the deep convolutional neural network to obtain the style encoding of the image;

[0034] 2. Use high-order attention mechanism to extract the content encoding of the image. Different from channel attention and spatial attention, which can only perform attention operation in one dimension, high-order attention mechanism can make the network model pay attention to the emotion-related features in the image in multiple dimensions, and learn more complete emotion-related content features in the image;

[0035] 3. Different from traditional static graph neural networks, dynamic graph convolutional neural networks adaptively generate specific graph association matrices, which can well capture the local correlation of emotional images;

[0036] 4. This method enhances the representation ability of image emotional features by fusing image style information, image content information and related information between image emotional labels, providing a new method for solving image emotional distribution prediction.

[0037] 5. Using the image sentiment distribution prediction results obtained above, output the matching sentiment image according to the sentiment polarity of the input word. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of an emotional image retrieval method based on image style coding;

[0039] Figure 2 A structural diagram of an emotional image retrieval device based on image style coding. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0041] Example 1

[0042] The embodiment of the present invention provides an emotional image retrieval method based on image style coding, see Figure 1 , the method comprises the following steps:

[0043] Step 101: Divide the visual emotion distribution dataset into a training set and a test set, and extract a multi-level feature map of the input image through a feature extraction network;

[0044] Step 102: The image style is related to the correlation between the low-dimensional feature maps. The correlation between the low-dimensional feature maps is calculated to obtain multiple GRAM matrices. These GRAM matrices are concatenated and then the style semantic features of the image are obtained through the convolutional neural network of style extraction.

[0045] Step 103: Image emotion not only depends on the style of the image, but is also closely related to the content features of the image. The feature map with higher dimensions contains rich content semantic features of the image. The multi-order attention mechanism is used to make the model focus on the feature areas in the image that are related to the emotion distribution, and obtain the content semantic features of the image.

[0046] Step 104: fusing the style semantic features and content semantic features of the image to obtain the emotional features of the image, and using the emotional features to encode and obtain a first emotional distribution prediction result;

[0047] Step 105: input the sentiment feature obtained in step 104 into the graph neural network with the sentiment category as the graph node, enhance the sentiment feature by using the correlation between the sentiment labels, and obtain the second sentiment distribution prediction result;

[0048] Step 106: fusing the first emotion distribution prediction result and the second emotion distribution prediction result into a final emotion distribution prediction result;

[0049] Step 107: using a natural language processing algorithm to calculate the sentiment polarity of the input search term, and outputting a sentiment image with a high sentiment distribution similarity according to the sentiment polarity of the input term.

[0050] Among them, the ResNet50 model is used to extract multi-level feature maps from images.

[0051] In step 102, an image style information encoding method based on a GRAM matrix is ​​adopted, which is expressed as follows:

[0052] F style =f style (Concat(G 0 ,G 1 ,G 2 ))…(1)

[0053] Among them, G k, k = 0, 1, 2 represents the GRAM style matrix calculated from the feature map output by the kth convolutional block; Concat(·) represents the concatenation of multiple levels of GRAM style matrices in the channel dimension; f style represents the image style encoding network, which consists of two convolutional blocks, each of which contains a convolutional layer, an instance normalization layer, and a pooling layer; F style Represents the final generated image style information encoding.

[0054] In step 103, the high-order attention calculation formula for focusing on different emotion-related feature regions in the image is as follows:

[0055]

[0056] Among them, A(x) represents the attention score at position x in the image, sigmoid(·) represents the activation function Sigmoid, is a learnable parameter, in T represents transpose, r represents the order, R is the highest order, c is the number of channels, d is the dth first-order tensor, is a real number space, z is the result of the action of different order descriptors, s is the order variable, u s is the decomposed vector and x is the local descriptor.

[0057] In step 104, the image content encoding obtained by each attention level is concatenated with the same image style encoding, and then the image emotion encoding of a specific category is obtained through the mapping network, as shown in the following formula:

[0058]

[0059] in, Represents the r-th order image content coding information; f ss (·) used to map the concatenated image style encoding and image content encoding into category-specific image emotion encoding; represents the category-specific image emotion encoding, F style Represents the style encoding of the image, and Concat means concatenating the two features in the channel dimension.

[0060] In step 105, the graph neural network for enhancing label relevance is composed of a static graph network and a dynamic graph network cascaded together, and the static graph network is shown in the following formula:

[0061]

[0062] Among them, A s is the adjacency matrix extracted from the training set, Ws represents the weight matrix of the static graph network, f SGCN Represents a static graph network.

[0063] The dynamic graph network is shown below:

[0064]

[0065] Among them, A d It is composed of a static graph network H r The adjacency matrix of the generated dynamic graph, A d =f d (H r ), W d is the weight matrix of the dynamic graph network, f d Generate a network for the dynamic adjacency matrix, H r is the node feature matrix output by the static graph network, f DGCN A dynamic graph network.

[0066] Example 2

[0067] The scheme in Example 1 is further introduced below in combination with specific calculation formulas and examples, as described below for details:

[0068] Step 201: extracting a multi-level visual feature map of an input image;

[0069] Among them, the ResNet50 network pre-trained on ImageNet is used to extract visual features of multiple dimensions of the input image.

[0070] Step 202: Calculate the correlation of feature maps in each dimension based on the GRAM matrix, use the deep convolutional neural network to calculate the correlation between feature maps of different dimensions, and generate the style encoding of the image; wherein, extract the feature maps output by the first convolution layer and the first two convolution blocks in step 201 Where k = 0, 1, 2 represents the kth output, X k Each feature map of is expanded into a vector Where m k =w k ×h k , n k Vectors are concatenated into a matrix Then use this matrix to calculate the GRAM matrix:

[0071]

[0072] G k Elements in yes and The inner product of is calculated as follows:

[0073]

[0074] in, is the matrix B k The ath element in the ith row of is the matrix B k The j-th row and the a-th element of , a is the column index of the element, w is the width of the feature map, h is the height of the feature map, n is the channel map of the feature map, and m is the product of the width and height of the feature map.

[0075] Since the dimensions of different feature maps are different, the sizes of the calculated GRAM matrices are different, so the three feature maps are unified to the same size through resampling. Since the GRAM matrix obtained from each layer of feature maps can only represent the style information at that level, and some style information spans different levels, the three GRAM matrices are concatenated together to obtain the cross-layer style feature map G. style =Concat(G 0 ,G 1 ,G 2 ), and then the image style encoding is obtained through the style encoding network:

[0076] F style =f style (G style ) (8)

[0077] Step 203: Obtain image content encoding using a multi-order attention mechanism;

[0078] Among them, the attention mechanism allows the neural network to pay more attention to the areas related to emotions in the image. The multi-order attention mechanism can extract features related to image emotions in multiple dimensions.

[0079] High-order attention defines a linear polynomial predictor for a position x in the image:

[0080]

[0081] where <·,·> is the inner product of the tensor, R is the order, is the r-order outer product of x, containing all r-order monomials of x. r is the weight of the r-order tensor combination. When r is large, w r Too many parameters will be introduced, causing overfitting problems. Through tensor decomposition and matrix transformation, we can get:

[0082]

[0083] When r = 1, is a matrix; C is the number of channels; d is the dth first-order tensor; is a learnable parameter implemented by 1×1 convolution; z r is the result of acting on the first-order descriptor.

[0084] When r>1, It represents the result of the action of descriptors of different orders.

[0085] According to the characteristics of convolutional neural networks, nonlinearity is introduced to obtain a high-order attention map of the local descriptor x:

[0086]

[0087] Among them, σ is a nonlinear activation function.

[0088] In the specific implementation, a 1×1 convolutional layer is used. In order to make different orders of attention focus on different visual features, adversarial loss is introduced:

[0089]

[0090] in, and denote the high-order attention networks of order a and b, respectively, and f adv Corresponding to a two-layer fully connected network, F content Encodes the image content.

[0091] Step 204: Generate a specific category of image emotion coding according to the image style coding and the image content coding;

[0092] Among them, step 202 and step 203 generate the image style code and multi-order image content code respectively, and the image content code obtained by each order of attention is spliced ​​with the same image style code, and then through the transformation network, the category-specific image emotion code is obtained, as shown in the following formula:

[0093]

[0094] in, represents the r-th order image content coding information, f ss Used to convert the concatenated image style encoding and image content encoding into category-specific image emotion encoding, Represents the category-specific image emotion encoding, where C represents the type of emotion. Find the maximum and average value of each feature map to obtain the predicted distribution of the middle-level image emotion:

[0095]

[0096] Among them, f mean is the average pooling operation, f maxis the maximum pooling operation, and λ is a hyperparameter used to balance average pooling and maximum pooling.

[0097] Step 205: utilizing the correlation between the emotion tags to enhance the final image emotion encoding through a graph convolutional neural network;

[0098] The graph convolutional neural network of this method consists of a static graph network and a dynamic graph network. The difference between them lies in the different ways of constructing the adjacency matrix. The dynamic graph network can adaptively generate a label adjacency matrix according to the input image to better capture the local correlation of the image.

[0099] The adjacency matrix of the static graph network is generated by the training set data. First, the training set labels are traversed, and the number of occurrences of all emotional labels is counted to obtain the label co-occurrence matrix. The items in the matrix that are less than the threshold ∈ are filtered out, and the conditional probability can be obtained, which is used as the adjacency matrix A of the static graph neural network. s .

[0100] Will Expand to l = w × h, which is used as the input of the static graph neural network, and the output H is obtained through the network r , the formula is as follows:

[0101]

[0102] in, Represents the r-order emotion encoding of the image.

[0103] The adjacency matrix of the dynamic graph network is calculated by the input node features. r The feature weights are obtained through the convolution layer, batch normalization layer, and global average pooling layer, and then the inner product with the feature matrix output by the static graph is calculated to obtain the adjacency matrix of the dynamic graph, which is expressed as follows:

[0104]

[0105] Among them, A d is the dynamic adjacency matrix, f d is the generative network of the adjacency matrix.

[0106] The output H of the static graph network r Through the dynamic graph network, the final output is obtained

[0107]

[0108] Then find the maximum and average value of the feature vector of each graph node to obtain the image sentiment prediction distribution of the graph neural network:

[0109]

[0110] The final image emotion prediction distribution of the embodiment of the present invention is obtained by weighting the image emotion prediction distribution output by the intermediate layer and the final image emotion prediction distribution:

[0111]

[0112] Among them, mean(·) is the average of the prediction results of each order. In order to supervise the final image emotion prediction distribution, the distribution prediction loss based on KL divergence is introduced:

[0113] L ldl =KLLoss(S,Q) (20)

[0114] Among them, Q is the sentiment distribution label of the input image.

[0115] The final objective function of the method proposed in the embodiment of the present invention is composed of sentiment distribution prediction loss and multi-order attention adversarial loss, which is expressed as follows:

[0116] L=L ldl +αL adv (twenty one)

[0117] Among them, α is a hyperparameter that balances the two losses.

[0118] The pre-trained model ALBERT of the natural language processing algorithm is used to obtain the sentiment polarity W of the input search term, and then find sentiment images with high similarity to this polarity based on the sentiment distribution of images in the image library, and output the sentiment images in descending order of similarity. Score = Corr(W,S i ),i=0,1,2,…,N (22)

[0119] Among them, S i represents the sentiment distribution of the i-th image in the image library, N represents the number of images in the image library, and Corr(·) represents the function for calculating the correlation between the input search term polarity and the image sentiment polarity.

[0120] Example 3

[0121] The embodiments of the present invention are described as follows:

[0122] We use 11,150 images from the Flickr website as the dataset. For each image, 11 staff members select the emotion they think the image expresses from 8 emotional tags, and count the 11 tag information as the emotional distribution of the image. The dataset is divided into a training set and a test set, using D train and D test express.

[0123] Among them, use I = {i1,i2,...,i N} and L={l1,l2,...,l N} represent the images and emotion labels of the training set or test set respectively. Each item in L corresponds to a vector of dimension 8, indicating the distribution of 8 emotions.

[0124] The network model was built using the PyTorch deep learning framework. The feature extraction network used the ResNet50 network model pre-trained with the ImageNet dataset. All other parameters were randomly initialized except for the static graph adjacency matrix generated by the training set data. The network model was trained end-to-end using the SGD optimizer until the model converged, and the predicted label distribution of the test set was obtained using the trained model.

[0125] There are currently six commonly used indicators for evaluating the effectiveness of network models, and their expressions are as follows:

[0126]

[0127]

[0128] d Cheb =max i |P i -Q i | (25)

[0129]

[0130]

[0131]

[0132] Among them, P and Q represent the predicted label distribution and the true label distribution respectively, m is the total number of labels in the training set, and d KL d Clk d Cheb and d Canb They represent KL divergence, Clark distance, Chebyshev distance, and Canberra distance, respectively, and are used to measure the difference between the predicted label distribution and the true label distribution. The smaller the value, the smaller the difference between the two distributions. Cos and S In They represent the Cosine and Intersection similarities respectively. The larger the value, the higher the similarity between the two distributions.

[0133] In summary, in order to obtain the emotion distribution information of the image, the embodiment of the present invention introduces image style coding to supplement the image content information to obtain a more complete image emotion feature coding. The correlation between emotion labels is introduced through the graph neural network, and the dynamic graph network adaptively generates a label association matrix to dynamically enhance the image emotion feature representation. The embodiment of the present invention combines multiple ideas such as image style information, attention mechanism, and graph convolutional neural network, which is particularly suitable for emotion distribution tasks with large image style differences and predicting image emotion distribution. The emotion polarity of the search term is obtained by pre-training the natural language processing network, and the correlation between the search term polarity and the image emotion distribution in the gallery is calculated to obtain images with a large emotion correlation with the search term as the output of the emotion image retrieval.

[0134] Example 4

[0135] An emotional image retrieval device based on image style coding, see Figure 2 The device includes: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to perform the following steps:

[0136] Based on multiple low-dimensional feature maps of the multi-level feature map, the correlation is calculated to obtain multiple GRAM matrices, and the GRAM matrices are concatenated and passed through the convolutional neural network for style extraction to obtain the style semantic features of the image;

[0137] Use a multi-order attention mechanism to focus on the feature areas in the image that are related to the emotional distribution, and obtain the content semantic features of the image;

[0138] The style semantic features and the content semantic features are integrated to obtain the emotional features of the image, and the emotional features are encoded to obtain the first emotional distribution prediction result;

[0139] The sentiment features are transmitted to the graph neural network as the input of the graph node, and the sentiment features are enhanced by using the correlation between the sentiment labels to obtain the second sentiment distribution prediction result;

[0140] The first sentiment distribution prediction result and the second sentiment distribution prediction result are merged into a final sentiment distribution prediction result;

[0141] A natural language processing algorithm is used to calculate the sentiment polarity of the input search terms, and an emotional image with a high degree of sentiment distribution similarity is output based on the sentiment polarity of the input terms.

[0142] Among them, the multi-level feature map is:

[0143] The visual emotion distribution dataset is divided into training set and test set, and the multi-level feature map of the input image is extracted through the feature extraction network.

[0144] Furthermore, the GRAM matrix is:

[0145] F style =f style (Concat(G 0 ,G 1 ,G 2 ))

[0146] Among them, G k , k = 0, 1, 2 represents the GRAM style matrix calculated from the feature map output by the kth convolutional block; Concat(·) represents the concatenation of multiple levels of GRAM style matrices in the channel dimension; f style represents the image style encoding network, which consists of two convolutional blocks, each of which contains a convolutional layer, an instance normalization layer, and a pooling layer; F style Represents the final generated image style information encoding.

[0147] The multi-order attention mechanism is used to focus on the feature areas in the image that are related to the emotional distribution, and the content semantic features of the image are obtained as follows:

[0148]

[0149] Where A(x) represents the attention score at position x in the image, sigmoid(·) represents the activation function, is a learnable parameter, in T represents transpose, r represents the order, R is the highest order, c is the number of channels, d is the dth first-order tensor, is a real number space, z is the result of the action of different order descriptors, s is the order variable, u s is the decomposed vector and x is the local descriptor.

[0150] Among them, the style semantic features and content semantic features are fused to obtain the emotional features of the image, and the first emotional distribution prediction result is obtained by encoding the emotional features:

[0151]

[0152] in, Represents the r-th order image content coding information; f ss (·) used to map the concatenated image style encoding and image content encoding into category-specific image emotion encoding; represents the category-specific image emotion encoding, F style Represents the style encoding of the image, and Concat means concatenating the two features in the channel dimension.

[0153] Furthermore, the graph neural network is composed of a static graph network and a dynamic graph network cascaded together.

[0154] The static graph network is as follows:

[0155]

[0156] Among them, A s is the adjacency matrix extracted from the training set, W s represents the weight matrix of the static graph network, f SGCN Represents a static graph network;

[0157] The dynamic graph network is as follows:

[0158]

[0159] Among them, A d It is composed of a static graph network H r The adjacency matrix of the generated dynamic graph, A d =f d (H r ), W d is the weight matrix of the dynamic graph network, f d Generate a network for the dynamic adjacency matrix, H r is the node feature matrix output by the static graph network, f DGCN A dynamic graph network.

[0160] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0161] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0162] The data signal is transmitted between the memory and the processor via a bus, which is not described in detail in the embodiment of the present invention.

[0163] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0164] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0165] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A sentiment image retrieval method based on image style coding, characterized in that: The method comprises the following steps: Based on multiple low-dimensional feature maps of the multi-level feature map, the correlation is calculated to obtain multiple GRAM matrices, and the GRAM matrices are concatenated and passed through the convolutional neural network for style extraction to obtain the style semantic features of the image; Use a multi-order attention mechanism to focus on the feature areas in the image that are related to the emotional distribution, and obtain the content semantic features of the image; The style semantic features and the content semantic features are integrated to obtain the emotional features of the image, and the emotional features are encoded to obtain the first emotional distribution prediction result; The sentiment features are transmitted to the graph neural network as the input of the graph node, and the correlation between the sentiment labels is used to enhance the sentiment features to obtain the second sentiment distribution prediction result; The first sentiment distribution prediction result and the second sentiment distribution prediction result are merged into a final sentiment distribution prediction result; A natural language processing algorithm is used to calculate the sentiment polarity of the input search terms, and an emotional image with a high degree of sentiment distribution similarity is output based on the sentiment polarity of the input terms; The style semantic features and the content semantic features are fused to obtain the emotional features of the image, and the first emotional distribution prediction result is obtained by encoding the emotional features: in, Represents the r-th order image content coding information; f ss (·) used to map the concatenated image style encoding and image content encoding into category-specific image emotion encoding; represents the category-specific image emotion encoding, F style Represents the style encoding of the image, and Concat means concatenating the two features in the channel dimension; The graph neural network is composed of a static graph network and a dynamic graph network cascaded together. The static graph network is as follows: Among them, A s is the adjacency matrix extracted from the training set, W s represents the weight matrix of the static graph network, f SGCN Represents a static graph network; The dynamic graph network is as follows: Among them, A d It is composed of a static graph network H r The adjacency matrix of the generated dynamic graph, A d =f d (H r ), W d is the weight matrix of the dynamic graph network, f d Generate a network for the dynamic adjacency matrix, H r is the node feature matrix output by the static graph network, f DGCN A dynamic graph network.

2. The emotional image retrieval method based on image style coding according to claim 1, characterized in that: The multi-level feature map is: The visual emotion distribution dataset is divided into training set and test set, and the multi-level feature map of the input image is extracted through the feature extraction network.

3. The emotional image retrieval method based on image style coding according to claim 1, characterized in that: The GRAM matrix is: F style =f style (Concat(G 0 ,G 1 ,G 2 )) Among them, G k , k = 0, 1, 2 represents the GRAM style matrix calculated from the feature map output by the kth convolutional block; Concat(·) represents the concatenation of multiple levels of GRAM style matrices in the channel dimension; f style represents the image style encoding network, which consists of two convolutional blocks, each of which contains a convolutional layer, an instance normalization layer, and a pooling layer; F style Represents the final generated image style information encoding.

4. The emotional image retrieval method based on image style coding according to claim 1, characterized in that: The multi-order attention mechanism is used to focus on the feature areas in the image that are related to the emotional distribution, and the content semantic features of the image are obtained as follows: Where A(x) represents the attention score at position x in the image, sigmoid(·) represents the activation function, is a learnable parameter, in T represents transpose, r represents the order, R is the highest order, c is the number of channels, d is the dth first-order tensor, is a real number space, z is the result of the action of different order descriptors, s is the order variable, u s is the decomposed vector and x is the local descriptor.

5. An emotional image retrieval device based on image style coding, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute the method steps described in any one of claims 1-4.

Images