A method of image sentiment analysis based on sentiment attribute mining

Abstract

The present invention belongs to the field of image content analysis, and discloses an image sentiment analysis method based on sentiment attribute mining. It utilizes user metadata information mining to construct a sentiment attribute subset with large semantic coverage and strong sentiment discriminability, and adopts a fusion neural network and a matrix decomposition model to predict a label matrix, thereby improving the accuracy and stability of the image sentiment analysis results. According to the characteristics of user label meta-information and sentiment attributes, four sentiment attribute characteristics are integrated and the concept selection model is optimized to complete the selection of the sentiment attribute subset; a convolutional neural network is used to extract visual features and optimize through visual content consistency; an adjacency graph guided by an external knowledge base is constructed and optimized through semantic relevance; image visual features and label semantic features are mapped to a potential shared space to achieve reconstruction of the predicted label matrix; finally, the trained reconstruction model is used to reconstruct the predicted label matrix, and a linear classifier is used for final sentiment recognition.

Description

Technical Field

[0001] The present invention relates to the field of image content analysis, and in particular to an image sentiment analysis method based on sentiment attribute mining. Background Art

[0002] Emotions have always been a psychological tool for people to adapt to survival. With the vigorous development of information science and technology and digital media technology, tens of thousands of electronic information data are transmitted on the Internet every day. More and more people will express their personal opinions and emotional tendencies in various forms on the Internet. Images have gradually become an important medium for Internet users to express their emotions. Analyzing the content of images to make abstract things concrete can not only serve as an auxiliary power for image retrieval and recommendation systems guided by user emotions; it can also combine big data with feedback mechanisms to provide auxiliary decision-making solutions for product design in terms of work creation; it can also perform automated, specific and real-time public opinion analysis and automatically identify problematic information. Therefore, how to perform image sentiment analysis has become a hot research issue that has attracted the attention of academia and industry in recent years.

[0003] Although the research results of image emotion recognition are quite rich, there are still many technical drawbacks:

[0004] First, most studies use empirical learning methods, which require a large amount of diverse training data, and have problems such as low efficiency, unexplainable process, and poor effect on small-scale tasks; second, there is a lack of mid-level semantic representation that can effectively reduce the semantic gap between image features and emotional semantics, and it is difficult to handle the complexity of the image emotion semantic mapping relationship, and the performance is not superior in fine-grained emotion recognition tasks; third, most studies use supervised learning methods, which require manual data annotation, and the amount of work required to obtain the data set is huge; fourth, in studies using Internet user images and user labels, the label information provided by users usually cannot meet the high-quality standards for visual content descriptions, and there are certain errors and omissions. Summary of the invention

[0005] In view of the above-mentioned defects in the existing image sentiment content analysis technology, the present invention proposes an image sentiment analysis method based on sentiment attribute mining. This method defines sentiment attributes as a group of visual concepts that contribute to the transmission of image sentiment, and uses user metadata information mining to build a concept selection model. The constrained quadratic linear optimization method is used to select sentiment attributes from the user-generated label set. A fusion neural network is further used to obtain the visual factor matrix and the semantic factor matrix, and a matrix decomposition model is combined to build a potential shared space, thereby reconstructing the prediction label matrix. This method can effectively improve the accuracy and stability of image sentiment analysis results.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An image sentiment analysis method based on sentiment attribute mining comprises the following steps:

[0008] Step 1: Based on the characteristics of user tags and sentiment attributes, a concept selection model that integrates the characteristics of four sentiment attributes is constructed, and the model is trained to complete the sentiment attribute subset selection.

[0009] Step 2: Use convolutional neural network to extract visual features, obtain the visual factor matrix by mapping the transformation matrix to the potential shared space, and optimize it through visual content consistency.

[0010] Step 3: Construct an adjacency graph guided by an external knowledge base, obtain the semantic factor matrix using a graph convolutional network, and optimize it through semantic relevance.

[0011] Step 4: Explore the correlation between the visual factor matrix and the semantic factor matrix through nonlinear measurement, build a reconstruction model of the predicted label matrix and train the model.

[0012] Step 5: Use the trained reconstruction model to reconstruct the prediction label matrix, and use a linear classifier for final emotion recognition.

[0013] Preferably, in the step 1, a method for constructing a concept selection model integrating four emotion attribute characteristics and obtaining an emotion attribute subset is used, where emotion attributes are defined as a group of visual semantic concepts that have a significant contribution to emotion transmission in the semantic part of the image cognitive layer. The four emotion attribute characteristics are semantic modelability, emotion discriminability, semantic coverage and finiteness, and their specific definitions and calculation methods are as follows:

[0014] 1) Semantic modelability: Using the information entropy calculation method, the intra-cluster difference is used as the probability weight of each cluster, and the inter-cluster difference is added to amplify the information entropy of the concept image cluster to calculate the semantic modelability. The specific calculation rules are as follows:

[0015]

[0016] Among them, ic k is the kth image cluster of concept c, p(ic k ) is the ratio of the number of images within a cluster to the number of representative images, intra_ds(IC) is the intra-cluster difference, and inter_ds(IC) is the inter-cluster difference;

[0017] 2) Emotion discriminability: Bayesian theorem is used to analyze the relationship between each concept and emotion, obtain the distribution of concepts in each type of emotional image, and calculate the emotion discriminability. The specific calculation rules are as follows:

[0018]

[0019] Where p(c|e) is the conditional probability of concept c under emotion category e, p(e) is the prior probability of emotion category, which is calculated by the ratio of the number of images in category e to the total number of images, and p(e|c) is the ability of concept c to distinguish emotion e.

[0020] 3) Semantic coverage: Semantic coverage is calculated based on mutual information. The specific calculation rules are as follows:

[0021]

[0022] where c i and c j are the i-th and j-th concepts in the concept space and the selected sentiment attribute set, respectively;

[0023] 4) Finiteness: The finiteness is calculated by the inverse of the set size. The specific calculation rules are as follows:

[0024]

[0025] 5) Convert the objective function into a maximum optimization problem with constraints, and transform it into a quadratic programming problem in matrix form. The main calculation rules are as follows:

[0026]

[0027] min s s T As,st0≤s≤1,1 T s=m

[0028] Among them, U s is a diagonal matrix whose elements u ii =SM(c i ), W s is a diagonal matrix whose elements w ii =DE(c i ), V s is a symmetric matrix whose element v ij =SC(c i ,c j ), s i ∈{1,0} represents concept c i Whether it is selected as the sentiment attribute to enter Θ, A = -{αU s +βW s +(1-α-β)(D V -V s )},D V is a diagonal matrix whose elements 1 is a vector of all 1s, and α and β are the allocation weights.

[0029] Preferably, the visual factor matrix learning method based on convolutional neural network in step 2 constructs a visual similarity graph Q based on K nearest neighbor method. U , and constrains the visual content based on the local geometric structure. The main calculation rules are as follows:

[0030]

[0031]

[0032]

[0033] Where σ is the bandwidth parameter, is a diagonal matrix, is the visual content consistency constraint.

[0034] Preferably, the step 3 specifically comprises:

[0035] 1) For the statistical adjacency graph A S and knowledge adjacency graph A K Perform matrix normalization to obtain and After combining, we get the adjacency graph A of user labels, and pass the threshold τ r To suppress the generation of redundant boundaries, the specific calculation rules are as follows:

[0036]

[0037]

[0038] Among them, η is the allocation weight, i, j are label indices, and the normalization operation is

[0039] 2) Based on the K nearest neighbor method, construct the semantic similarity graph Q V , and constrains the semantic content based on the local geometric structure. The main calculation rules are as follows:

[0040]

[0041]

[0042]

[0043] in, is a diagonal matrix, is the visual content consistency constraint.

[0044] Preferably, in the prediction label matrix reconstruction method for exploring the correlation between the factor matrix composed of the image and label information by a nonlinear measurement method, the step 4 specifically comprises:

[0045] 1) The loss function is calculated by integrating the sparse noise constraint, visual factor matrix acquisition and semantic factor matrix acquisition. The main calculation rules are as follows:

[0046]

[0047] Among them, α1, β1, β2, λ1, λ2 are the assigned weights, P is the predicted label matrix, G is the user label matrix, and W U is the visual transformation matrix, S is the visual feature matrix, V is the semantic factor matrix, W V is the semantic transformation matrix;

[0048] 2) By introducing the auxiliary matrix Transform and simplify the objective function, and its main calculation rules are as follows:

[0049]

[0050]

[0051] Among them, P * =2U T V-1, The sgn(·) function outputs +1 for positive values ​​and -1 for negative values.

[0052] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages:

[0053] 1. The present invention is based on multi-feature modeling, so that the emotional feature set has the advantages of semantic modeling, emotional discrimination, large semantic coverage and limited number;

[0054] 2. The present invention performs sentiment mining based on user metadata information, which solves the drawbacks of the middle-level semantic concepts currently defined for image sentiment analysis. The obtained middle-level semantic concepts have a large semantic coverage and strong sentiment discriminability;

[0055] 3. The present invention uses user-generated images and associated tags for prediction, which reduces the cost of obtaining data for supervised learning methods in computer vision;

[0056] 4. The present invention proposes a learning method for a semantic factor matrix based on a graph convolutional network. By introducing an external knowledge base to guide the construction of an adjacency graph, the negative impact of noise association information between labels is weakened, and a more accurate prediction label matrix is ​​obtained;

[0057] 5. The present invention maximizes the flexibility of the matrix decomposition algorithm by integrating the neural network and the matrix decomposition model, and effectively improves the accuracy and stability of the image sentiment analysis results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of an image sentiment analysis method based on sentiment attribute mining provided by the present invention.

[0059] Figure 2 It is a framework diagram of a fusion neural network and matrix decomposition model for predicting a label matrix provided by the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0061] Embodiment 1

[0062] A method for image sentiment analysis based on sentiment attribute mining uses user metadata information mining to build a concept selection model to select sentiment attributes. It further proposes a fusion neural network and matrix decomposition model to predict image labels based on user annotations. The operation steps are as follows:

[0063] Step 1: Based on the characteristics of user tags and sentiment attributes, a concept selection model integrating the characteristics of four sentiment attributes is constructed, and the model is trained to complete the selection of sentiment attribute subsets;

[0064] Step 2: Use convolutional neural networks to extract visual features, obtain the visual factor matrix by mapping the transformation matrix to the potential shared space, and optimize it through visual content consistency;

[0065] Step 3: Construct an adjacency graph guided by an external knowledge base, obtain the semantic factor matrix using a graph convolutional network, and optimize it through semantic relevance;

[0066] Step 4: Explore the correlation between the visual factor matrix and the semantic factor matrix through nonlinear measurement, and build a reconstruction model for the predicted label matrix;

[0067] Step 5: Use the trained reconstruction model to reconstruct the prediction label matrix, and use a linear classifier for final emotion recognition.

[0068] This embodiment is based on the image sentiment analysis method of sentiment attribute mining, defines the concept of sentiment attribute, utilizes user information mining, constructs a concept selection model to select sentiment attributes, further adopts a fusion neural network to obtain the visual factor matrix and the semantic factor matrix, combines the matrix decomposition model, and reconstructs the prediction label matrix. The method of this embodiment can effectively improve the accuracy and stability of the image sentiment analysis results.

[0069] Embodiment 2:

[0070] This embodiment is basically the same as the above embodiment, with the following features:

[0071] In this embodiment, the method of constructing a concept selection model integrating four emotional attribute characteristics and obtaining an emotional attribute subset in step 1 defines emotional attributes as a group of visual semantic concepts that have a significant contribution to emotional transmission in the semantic part of the image cognitive layer. The four emotional attribute characteristics are semantic modelability, emotional discriminability, semantic coverage and finiteness. The specific definition and calculation method are as follows:

[0072] 1) Semantic modelability: Using the information entropy calculation method, the intra-cluster difference is used as the probability weight of each cluster, and the inter-cluster difference is added to amplify the information entropy of the concept image cluster to calculate the semantic modelability. The specific calculation rules are as follows:

[0073]

[0074] Among them, ic k is the kth image cluster of concept c, p(ic k ) is the ratio of the number of images within a cluster to the number of representative images, intra_ds(IC) is the intra-cluster difference, and inter_ds(IC) is the inter-cluster difference;

[0075] 2) Emotion discriminability: Bayesian theorem is used to analyze the relationship between each concept and emotion, obtain the distribution of concepts in each type of emotional image, and calculate the emotion discriminability. The specific calculation rules are as follows:

[0076]

[0077] Where p(c|e) is the conditional probability of concept c under emotion category e, p(e) is the prior probability of emotion category, which is calculated by the ratio of the number of images in category e to the total number of images, and p(e|c) is the ability of concept c to distinguish emotion e.

[0078] 3) Semantic coverage: Semantic coverage is calculated based on mutual information. The specific calculation rules are as follows:

[0079]

[0080] where c i and c j are the i-th and j-th concepts in the concept space and the selected sentiment attribute set, respectively;

[0081] 4) Finiteness: The finiteness is calculated by the inverse of the set size. The specific calculation rules are as follows:

[0082]

[0083] 5) Convert the objective function into a maximum optimization problem with constraints, and transform it into a quadratic programming problem in matrix form. The main calculation rules are as follows:

[0084]

[0085] min s s T As,st0≤s≤1,1 T s=m

[0086] Among them, U s is a diagonal matrix whose elements u ii =SM(c i ), W s is a diagonal matrix whose elements w ii =DE(c i ), V s is a symmetric matrix whose element v ij =SC(c i ,c j ), s i ∈{1,0} represents concept c i Whether it is selected as the sentiment attribute to enter Θ, A = -{αU s +βW s +(1-α-β)(D V -V s )},D V is a diagonal matrix whose elements 1 is a vector of all 1s, and α and β are the allocation weights.

[0087] In this embodiment, the visual factor matrix learning method based on convolutional neural network in step 2 constructs a visual similarity graph Q based on the K nearest neighbor method. U , and constrains the visual content based on the local geometric structure. The main calculation rules are as follows:

[0088]

[0089]

[0090]

[0091] Where σ is the bandwidth parameter, is a diagonal matrix, is the visual content consistency constraint.

[0092] In this embodiment, step 3 specifically includes:

[0093] 1) For the statistical adjacency graph A S and knowledge adjacency graph A K Perform matrix normalization to obtain and After combining, we get the adjacency graph A of user labels, and pass the threshold τ r To suppress the generation of redundant boundaries, the specific calculation rules are as follows:

[0094]

[0095]

[0096] Among them, η is the allocation weight, i, j are label indices, and the normalization operation is

[0097] 2) Based on the K nearest neighbor method, construct the semantic similarity graph Q V , and constrains the semantic content based on the local geometric structure. The main calculation rules are as follows:

[0098]

[0099]

[0100]

[0101] in, is a diagonal matrix, is the visual content consistency constraint.

[0102] In this embodiment, the prediction label matrix reconstruction method for exploring the correlation between the factor matrix composed of image and label information by nonlinear measurement method, the step 4 specifically includes:

[0103] 1) The loss function is calculated by integrating the sparse noise constraint, visual factor matrix acquisition and semantic factor matrix acquisition. The main calculation rules are as follows:

[0104]

[0105] Among them, α1, β1, β2, λ1, λ2 are the assigned weights, P is the predicted label matrix, G is the user label matrix, and W U is the visual transformation matrix, S is the visual feature matrix, V is the semantic factor matrix, WV is the semantic transformation matrix;

[0106] 2) By introducing the auxiliary matrix Transform and simplify the objective function, and its main calculation rules are as follows:

[0107]

[0108]

[0109] Among them, P * =2U T V-1, The sgn(·) function outputs +1 for positive values ​​and -1 for negative values.

[0110] This embodiment is an image sentiment analysis method based on sentiment attribute mining, which defines sentiment attributes as a group of visual concepts that contribute to the transmission of image sentiment. It constructs a concept selection model by mining user metadata information, selects sentiment attributes from a user-generated label set through a constrained quadratic linear optimization method, further uses a fusion neural network to obtain a visual factor matrix and a semantic factor matrix, and combines a matrix decomposition model to construct a potential shared space, thereby reconstructing a predicted label matrix. This method can effectively improve the accuracy and stability of image sentiment analysis results.

[0111] Embodiment three:

[0112] This embodiment is basically the same as the above embodiment, with the following features:

[0113] Figure 1 The flowchart of the image sentiment analysis method based on sentiment attribute mining disclosed in the present invention is shown as follows: Figure 1 As shown, the method comprises the following steps:

[0114] Step 1: Based on the characteristics of user tags and sentiment attributes, a concept selection model that integrates the characteristics of four sentiment attributes is constructed, and the model is trained to complete the sentiment attribute subset selection, which specifically includes the following steps:

[0115] 1) Calculate the semantic modelability through the intra-cluster visual difference and inter-cluster visual difference of the associated image clusters of each concept. The specific steps are as follows:

[0116] 1-1) For each concept c in the concept set C, the associated label feature set of the image Calculate c and the i image Calculate the semantic similarity between concept c and the i-th image. image Semantic similarity scores of the images The calculation formula is as follows:

[0117]

[0118] Among them, d(·) calculates the cosine similarity between the input vectors, It is the tth image According to the calculated semantic similarity scores, the first K images with the highest semantic similarity are selected as the representative image set of each concept.

[0119] 1-2) Predefine the number of image clusters as k c Based on the similarity between image features, K-means clustering is used to obtain k c The image set IC of image clusters is discarded, and the number of image clusters contained in the image set of each concept is finally obtained as |IC|. The visual feature distance between the images in each cluster is used to calculate the intra-cluster visual difference intra_ds(ic k ), the calculation process is as follows:

[0120]

[0121] Among them, ic k is the kth image cluster of concept c, Calculate the i-th image and jth image The cosine distance of image features.

[0122] 1-3) Obtained by performing average pooling operation on the image features within the cluster It is the average vector of all visual features in the kth image cluster. The cosine distance of the average visual features of each cluster in the image cluster set is calculated to obtain the inter-cluster difference inter_ds (IC). The calculation process is as follows:

[0123]

[0124] 1-4) Based on the calculation method of information entropy, the intra-cluster difference is used as the probability weight of each cluster, and the inter-cluster difference is added to amplify the information entropy of the concept image cluster to calculate the final semantic modelability. The calculation process is as follows:

[0125]

[0126] Among them, p(ic k ) is the ratio of the number of images in the cluster to the number of representative images.

[0127] 1-5) Calculate the sum of the semantic modelability of the selected concepts to obtain the semantic modelability result of the sentiment attribute set. The calculation process is as follows:

[0128]

[0129] 2) Use Bayesian theorem to analyze the relationship between each concept and emotion, obtain the distribution of concepts in each type of emotional image, and calculate the emotional discriminability. The specific steps are as follows:

[0130] 2-1) Calculate the conditional probability p(c|e) of concept c under sentiment category e. The calculation process is as follows:

[0131]

[0132] in, represents the number of images belonging to emotion category e, represents the total number of images whose associated user label has concept c and belongs to sentiment category e.

[0133] 2-2) Calculate the distribution p(c) of each concept. The calculation process is as follows:

[0134]

[0135] 2-3) Based on Bayes’ theorem, calculate the probability p(e|c) that the sentiment classification of this image is e when the concept c exists. The calculation process is as follows:

[0136]

[0137] Among them, the prior probability p(e) of the emotion category is the ratio of the number of images belonging to category e to the total number of images, and p(e|c) represents the ability of concept c to distinguish emotion e.

[0138] 2-4) Based on information entropy, the final emotional discriminability score ED(c) is calculated. The calculation process is as follows:

[0139]

[0140] 2-5) Calculate the sum of the sentiment discriminability scores of the selected concepts to obtain the sentiment discriminability score of the sentiment attribute set. The calculation process is as follows:

[0141]

[0142] 3) Calculate semantic coverage based on mutual information. In order to avoid redundant features of repeated and similar labels for image sentiment analysis, and to make the sentiment attribute set cover the entire semantic concept space as much as possible, the concepts in the selected sentiment attributes should have a low semantic correlation, that is, semantic coverage. The calculation process is as follows:

[0143]

[0144] where c i and cj are the i-th and j-th concepts in the concept space and the selected sentiment attribute set, respectively.

[0145] 4) Calculate finiteness. In order to avoid the dimensionality disaster in the emotion recognition task, it is necessary to limit the number of emotion attributes. The finiteness score is expressed by the size of the set. The more concepts a set contains, the lower its finiteness score. The calculation process of the finiteness score L(Θ) of the emotion attribute set is as follows:

[0146]

[0147] 5) Integrate the four emotional attribute characteristic calculation formulas, transform the optimization goal of the concept selection model into a quadratic programming problem, solve this quadratic programming problem, and obtain the emotional attribute subset. The specific steps are as follows:

[0148] 5-1) Adding weight terms to fuse the four emotional attribute characteristic calculation formulas, the result obtained is the optimization target of the concept selection model, and the results are as follows:

[0149] max Θ {αSM(Θ)+βDE(Θ)+γSC(Θ;C)+(1-α-β-γ)L(Θ)}

[0150] Among them, α, β, and γ are weight distribution parameters used to balance the four characteristics.

[0151] 5-2) By fixing the target sentiment attribute subset size |Θ| to a constant m, the objective function is transformed into a maximum optimization problem with constraints. Define the diagonal matrix U s and element u ii =SM(c i ), the diagonal matrix W s and element w ii =DE(c i ), symmetric matrix V s and element v ij =SC(c i ,c j ), and a vector s, where s i ∈{1,0} represents concept c i Whether it is selected as the sentiment attribute to enter Θ, according to the above definition, the optimization problem is transformed into the following formula:

[0152]

[0153]

[0154] 5-3) By converting the optimization problem into a matrix form, a quadratic programming problem is obtained. The specific results are as follows:

[0155] mins s T As,st0≤s≤1,1 T s=m

[0156] Where A = -{αU s +βW s +(1-α-β)(D V -V s )},D V is a diagonal matrix whose elements 1 is a vector of all 1s.

[0157] 5-4) Solve this quadratic programming problem, obtain the vector s, and select the m concepts with the highest scores corresponding to s to form a sentiment attribute subset.

[0158] Step 2: Use convolutional neural networks to extract visual features, obtain the visual factor matrix by mapping the transformation matrix to the potential shared space, and optimize it through visual content consistency. Figure 2 The framework diagram of the fusion neural network and matrix decomposition model for predicting the label matrix provided by the present invention is shown, wherein the visual feature part shows the process of the visual factor matrix learning method based on convolutional neural network provided by the present invention.

[0159] Combine the following Figure 2 To be more specific:

[0160] 1) For the input image matrix X, use the convolutional neural network model f(·) to obtain the visual feature matrix S = f(X), and use the visual transformation matrix W U , map the visual feature matrix S to the potential shared space to obtain the visual factor matrix U, which is calculated as follows:

[0161] U=W U S

[0162] 2) Optimize the visual factor matrix and constrain the visual factor vector through visual content consistency.

[0163] 2-1) Based on the K nearest neighbor method, construct a visual similarity graph Q U , the calculation process is as follows:

[0164]

[0165] Among them, σ is the bandwidth parameter.

[0166] 2-2) Optimize the visual factor matrix and calculate the visual content consistency constraint item based on the local geometric structure constraint. The calculation process is as follows:

[0167]

[0168] in, is a diagonal matrix, Convert the above formula into matrix form, and we can get the formula:

[0169]

[0170] in, is the graph Laplacian matrix of the visual similarity graph, defined as D stands for The degree matrix of .

[0171] Step 3: Construct an adjacency graph guided by an external knowledge base, obtain the semantic factor matrix using a graph convolutional network, and optimize it through semantic relevance. Figure 2 The framework diagram of the fusion neural network and matrix decomposition model for predicting the label matrix provided by the present invention is shown, wherein the semantic feature part shows the process of the semantic factor matrix learning method based on graph convolutional network provided by the present invention.

[0172] Combine the following Figure 2 To be more specific:

[0173] 1) Combine the statistical adjacency graph and the knowledge adjacency graph to construct an adjacency graph guided by the external knowledge base. The specific steps are as follows:

[0174] 1-1) Calculate the co-occurrence information of user tags in a given image dataset to construct a statistical adjacency graph Using the threshold τ s Filter the noise edges and construct a statistical adjacency graph. The construction process is as follows:

[0175]

[0176] Among them, P(c j |c i ) is a given label c i Time label c j Conditional probability of occurrence. Adjust the elements in the statistical adjacency graph to avoid the over-smoothing problem caused by the graph convolutional network. The calculation process is as follows:

[0177]

[0178] Where ρ is the allocation weight.

[0179] 1-2) Constructing a knowledge adjacency graph by extracting associated knowledge from ConceptNet The build process is as follows:

[0180]

[0181] Among them, R ijRepresents the association relationship between the word vertices of ConceptNet where two labels are located, and w is the association weight.

[0182] 1-3) For the statistical adjacency graph A S and knowledge adjacency graph A K Perform matrix normalization to obtain and The normalization operation is The statistical adjacency graph and the knowledge adjacency graph are summed based on the weight η to obtain the adjacency graph A of the user label. The calculation process is as follows:

[0183]

[0184] By threshold τ r To suppress the generation of redundant boundaries, the calculation process is as follows:

[0185]

[0186] 1-4) The adjacency graph A and the label feature matrix T obtained above are used as the input of the GCN model, and the final label semantic feature matrix is ​​obtained through multi-layer graph convolution operations. The calculation form is as follows:

[0187]

[0188] Among them, θ g is the graph convolution network parameter, and h(·) is the graph convolution operation.

[0189] 2) Through a semantic transformation matrix W V Map the output node features Z of the graph convolutional network to the potential shared space to obtain the semantic factor matrix V:

[0190] V=W V Z T

[0191] 3) Optimize the semantic factor matrix and constrain the semantic factor vector through semantic relevance. The specific steps are as follows:

[0192] 3-1) Based on the K nearest neighbor method, construct the semantic similarity graph Q V , which is expressed as:

[0193]

[0194] Where σ is the bandwidth parameter.

[0195] 3-2) Based on the local geometric structure constraints, the semantic content consistency constraint items are calculated. The calculation process is as follows:

[0196]

[0197] in, is a diagonal matrix, Convert the above formula into matrix form, and we can get the formula:

[0198]

[0199] in, is the graph Laplacian matrix of the semantic similarity graph, defined as D stands for The degree matrix of .

[0200] Step 4: Explore the correlation between the visual factor matrix and the semantic factor matrix through nonlinear measurement, build a reconstruction model of the predicted label matrix and train the model. Specifically, the reconstruction model is optimized through visual content consistency constraints and semantic relevance constraints. Figure 2 The fusion part shows the flow chart of the prediction label matrix reconstruction technology based on the matrix decomposition model provided by the present invention.

[0201] Combine the following Figure 2 To be more specific:

[0202] 1) Construct the objective function of the model. The model reconstructs the original matrix through two factor matrices. The objective function is as follows:

[0203]

[0204] Where P is the predicted label matrix.

[0205] 2) Through the loss function Optimize the model. The specific steps are as follows:

[0206] 2-1) By subtracting the given user label matrix G from the noise label matrix E, the predicted label matrix P = GE is obtained. The optimized calculation process is as follows:

[0207]

[0208] 2-2) The loss function is calculated by integrating the sparse noise constraint, visual factor matrix acquisition and semantic factor matrix acquisition. The calculation process is as follows:

[0209]

[0210] Among them, α1, β1, β2, λ1, λ2 are the allocation weights.

[0211] 2-3) Use an iterative optimization algorithm to update the gradient of each variable by updating one variable and fixing the other variables. For the visual transformation matrix W from visual features to the potential shared spaceU , the process of calculating the gradient of each update is as follows:

[0212]

[0213]

[0214] Where η is the learning rate. For the semantic transformation matrix W from semantic features to the potential shared space V , the process of calculating the gradient of each update is as follows:

[0215]

[0216]

[0217] The P value is updated through the two updated gradient results.

[0218] 2-4) By introducing the auxiliary matrix The objective function is transformed to obtain the closed-form solution of the above problem. The transformation process is as follows:

[0219]

[0220] Where P * =2U T V-1, The objective function is simplified by mathematical derivation, and the simplification process is as follows:

[0221]

[0222]

[0223]

[0224]

[0225] The sgn(·) function outputs +1 for positive values ​​and -1 for negative values.

[0226] 3) Train the reconstruction model. Based on the trained model, the predicted label matrix P can be obtained, and the labels corresponding to the top 5 elements are used as the predicted labels for each image sample.

[0227] Step 5: For the test samples, the trained reconstruction model is used to reconstruct the prediction label matrix, and a linear classifier is used for the final emotion recognition.

[0228] The above-mentioned embodiment is based on the image sentiment analysis method of sentiment attribute mining, and utilizes user metadata information mining to construct a sentiment attribute subset with large semantic coverage and strong sentiment discriminability, and adopts a fusion neural network and matrix decomposition model to predict the label matrix, thereby improving the accuracy and stability of the image sentiment analysis results. According to the characteristics of user label meta-information and sentiment attributes, four sentiment attribute characteristics are integrated and the concept selection model is optimized to complete the selection of the sentiment attribute subset; the convolutional neural network is used to extract visual features and optimize through visual content consistency; an adjacency graph guided by an external knowledge base is constructed and optimized through semantic relevance; the image visual features and label semantic features are mapped to the potential shared space to realize the reconstruction of the predicted label matrix; finally, the trained reconstruction model is used to reconstruct the predicted label matrix, and a linear classifier is used for the final sentiment recognition.

[0229] The above describes the embodiments of the present invention in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made according to the purpose of the invention. Any changes, modifications, substitutions, combinations or simplifications made according to the spirit and principle of the technical solution of the present invention should be equivalent replacement methods. As long as they meet the purpose of the invention and do not deviate from the technical principles and inventive concepts of the present invention, they belong to the protection scope of the present invention.

Claims

1. An image sentiment analysis method based on sentiment attribute mining, which uses user metadata information mining to build a concept selection model to achieve sentiment attribute selection, and further proposes a fusion neural network and matrix decomposition model to achieve image label prediction based on user annotations, characterized in that: The steps are as follows: Step 1: Based on the characteristics of user tags and sentiment attributes, a concept selection model integrating the characteristics of four sentiment attributes is constructed, and the model is trained to complete the selection of sentiment attribute subsets; Step 2: Use convolutional neural networks to extract visual features, obtain the visual factor matrix by mapping the transformation matrix to the potential shared space, and optimize it through visual content consistency; Step 3: Construct an adjacency graph guided by an external knowledge base, obtain the semantic factor matrix using a graph convolutional network, and optimize it through semantic relevance; Step 4: Explore the correlation between the visual factor matrix and the semantic factor matrix through nonlinear measurement, and build a reconstruction model for the predicted label matrix; Step 5: Use the trained reconstruction model to reconstruct the prediction label matrix, and use the linear classifier for the final emotion recognition; In the step 1, a method for constructing a concept selection model integrating four emotional attribute characteristics and obtaining an emotional attribute subset is used. The emotional attribute is defined as a group of visual semantic concepts that have a significant contribution to emotional transmission in the semantic part of the image cognitive layer. The four emotional attribute characteristics are semantic modelability, emotional discriminability, semantic coverage and finiteness. The specific definition and calculation method are as follows: 1) Semantic modelability: Using the information entropy calculation method, the intra-cluster difference is used as the probability weight of each cluster, and the inter-cluster difference is added to amplify the information entropy of the concept image cluster to calculate the semantic modelability. The specific calculation rules are as follows: Among them, ic k is the kth image cluster of concept c, p(ic k ) is the ratio of the number of images within a cluster to the number of representative images, intra_ds(IC) is the intra-cluster difference, and inter_ds(IC) is the inter-cluster difference; 2) Emotion discriminability: Bayesian theorem is used to analyze the relationship between each concept and emotion, obtain the distribution of concepts in each type of emotional image, and calculate the emotion discriminability. The specific calculation rules are as follows: Where p(c|e) is the conditional probability of concept c under emotion category e, p(e) is the prior probability of emotion category, which is calculated by the ratio of the number of images in category e to the total number of images, and p(e|c) is the ability of concept c to distinguish emotion e. 3) Semantic coverage: Semantic coverage is calculated based on mutual information. The specific calculation rules are as follows: where c i and c j are the i-th and j-th concepts in the concept space and the selected sentiment attribute set, respectively; 4) Finiteness: The finiteness is calculated by the inverse of the set size. The specific calculation rules are as follows: 5) Convert the objective function into a maximum optimization problem with constraints, and transform it into a quadratic programming problem in matrix form. The main calculation rules are as follows: min s s T As,s.t.0≤s≤1,1 T s=m Among them, U s is a diagonal matrix whose elements u ii =SM(c i ), W s is a diagonal matrix whose elements w ii =DE(c i ), V s is a symmetric matrix whose element v ij =SC(c i ,c j ), s i ∈{1,0} represents concept c i Whether it is selected as the sentiment attribute to enter Θ, A = -{αU s +βW s +(1-α-β)(D V -V s )},D V is a diagonal matrix whose elements 1 is a vector of all 1s, α and β are the allocation weights; The step 4 specifically comprises: 1) The loss function is calculated by integrating the sparse noise constraint, visual factor matrix acquisition and semantic factor matrix acquisition. The main calculation rules are as follows: Among them, α1, β1, β2, λ1, λ2 are the assigned weights, P is the predicted label matrix, G is the user label matrix, and W U is the visual transformation matrix, S is the visual feature matrix, V is the semantic factor matrix, W V is the semantic transformation matrix; 2) By introducing the auxiliary matrix Transform and simplify the objective function, and its main calculation rules are as follows: Among them, P * =2U T V-1, The sgn(·) function outputs +1 for positive values ​​and -1 for negative values.

2. The image sentiment analysis method based on sentiment attribute mining according to claim 1 is characterized in that: The visual factor matrix learning method based on convolutional neural network in step 2 constructs the visual similarity graph Q based on the K nearest neighbor method. U , and constrains the visual content based on the local geometric structure. The main calculation rules are as follows: Where σ is the bandwidth parameter, is a diagonal matrix, is the visual content consistency constraint.

3. The image sentiment analysis method based on sentiment attribute mining according to claim 1 is characterized in that: The step 3 specifically comprises: 1) For the statistical adjacency graph A S and knowledge adjacency graph A K Perform matrix normalization to obtain and After combining, we get the adjacency graph A of user labels, and pass the threshold τ r To suppress the generation of redundant boundaries, the specific calculation rules are as follows: Among them, η is the allocation weight, i, j are label indices, and the normalization operation is 2) Based on the K nearest neighbor method, construct the semantic similarity graph Q V , and constrains the semantic content based on the local geometric structure. The main calculation rules are as follows: in, is a diagonal matrix, is the visual content consistency constraint.

Images