Image fine-grained sentiment prediction method, electronic device and storage medium

Abstract

Embodiments of the present invention provide a fine-grained image sentiment prediction method, the method comprising: grouping multiple sentiment words describing the sentiment of an image based on the correlation between sentiment words to obtain multiple sentiment word sets, wherein each sentiment word set has a representative sentiment word; predicting the representative sentiment value corresponding to each representative sentiment word of a target image; and predicting the sentiment values ​​corresponding to the remaining sentiment words in the sentiment word set of the target image based on the representative sentiment values ​​of the representative sentiment words of each sentiment word set. Another embodiment of the present invention also provides an electronic device and a computer-readable storage medium.

Description

Technical Field

[0001] This invention relates to the field of image sentiment prediction technology, and more particularly to a small-sample, fine-grained image sentiment prediction method, specifically to an image fine-grained sentiment prediction method, electronic device, and storage medium. Background Technology

[0002] With the rapid development of multimedia and the internet industries, people tend to upload images to various online platforms to share their experiences and feelings. Images, as a medium, contain more information than text; they transcend cultural and temporal barriers more easily, conveying richer emotional information and artistic conceptions. Detecting image data, obtaining relevant information, and establishing meaningful content descriptions are important research areas in image understanding and computer vision. How to interpret and extract the emotional information contained in images has become a research hotspot in recent years. From the user's experience perspective, the categories of emotions can be diverse, especially for images related to artistic expression, such as abstract paintings, film stills, and art illustrations.

[0003] In the process of realizing the concept of this invention, the inventors found that there are at least the following problems in the related technologies: the image emotion prediction in the related technologies often adopts positive and negative classification of emotion or 6 or 8 types of emotion based on basic human emotions, and the given information is mostly emotion labels, lacking effective prediction of the degree of emotion arousal. Summary of the Invention

[0004] In view of this, the technical problem to be solved by the present invention is to provide a method, device and medium for fine-grained image emotion prediction, which solves the problem that the degree of emotional arousal of an image cannot be effectively predicted in the related art.

[0005] To address the aforementioned technical problems, specific embodiments of the present invention provide a fine-grained image sentiment prediction method, comprising: grouping multiple sentiment words describing image sentiment based on the correlation between sentiment words to obtain multiple sentiment word sets, wherein each sentiment word set has a representative sentiment word; predicting the representative sentiment value corresponding to each representative sentiment word of the target image; and predicting the sentiment value corresponding to the remaining sentiment words in the sentiment word set of the target image based on the representative sentiment value of the representative sentiment word of each sentiment word set.

[0006] According to an optional embodiment of the present invention, the step of grouping multiple sentiment words describing the sentiment of an image based on the correlation between sentiment words to obtain multiple sentiment word sets includes: obtaining a factor loading matrix of multiple sentiment words using principal component analysis, wherein the factor loading matrix is ​​composed of factor loading values; determining multiple factors and multiple sentiment word sets according to the factor loading matrix, wherein the factor corresponding to the maximum factor loading value of multiple sentiment words corresponds one-to-one with the sentiment word; and determining the representative sentiment word of each sentiment word set using a random forest algorithm.

[0007] According to an optional embodiment of the present invention, the step of determining the representative sentiment word for each sentiment word set includes: constructing an initial image sentiment prediction model for each sentiment word using a random forest algorithm; determining the prediction accuracy of the sentiment words in each sentiment word set based on the initial image sentiment prediction model; and determining the representative sentiment word for each sentiment word set based on the prediction accuracy.

[0008] According to an optional embodiment of the present invention, the step of predicting the representative sentiment value corresponding to each representative sentiment word in a target image includes: extracting multidimensional initial image features of the target image; for each sentiment word, performing dimensionality reduction on the multidimensional initial image features based on a combination of minimum absolute shrinkage selection operator and random forest feature importance to obtain dimensionality-reduced image features; and predicting the representative sentiment value corresponding to each representative sentiment word based on the dimensionality-reduced image features.

[0009] According to an optional embodiment of the present invention, the initial image features include low-level features, high-level features, and key region features. The low-level features include color features and texture features. The high-level features include aesthetic features and composition features. The key region features include color moment features and shape contrast features.

[0010] According to an optional embodiment of the present invention, the step of predicting the representative sentiment value corresponding to each representative sentiment word based on dimensionality-reduced image features includes: establishing an image sentiment prediction model corresponding to each representative sentiment word based on an ensemble learning method; and inputting the dimensionality-reduced image features into the image sentiment prediction model corresponding to each representative sentiment word to obtain the representative sentiment value corresponding to the representative sentiment word.

[0011] According to an optional embodiment of the present invention, the step of establishing an image sentiment prediction model corresponding to each representative sentiment word based on an ensemble learning method includes: constructing a scene image library containing multiple images and covering all original sentiments; using most of the images in the scene image library as an initial accuracy training set and the remaining images in the scene image library as an initial accuracy test set; and training the image sentiment prediction model corresponding to each representative sentiment word using the initial accuracy training set and the initial accuracy test set, thereby constructing the image sentiment prediction model corresponding to each representative sentiment word.

[0012] According to an optional embodiment of the present invention, the step of inputting dimensionality-reduced image features into an image sentiment prediction model corresponding to each representative sentiment word to obtain the representative sentiment value corresponding to that representative sentiment word includes: inputting the dimensionality-reduced image features into each first-level learner of the image sentiment prediction model to obtain a first-level prediction result corresponding to that first-level learner, wherein each image sentiment prediction model includes multiple first-level learners, multiple second-level learners, and a linear regression model; inputting the first-level prediction results output by all first-level learners into each second-level learner to obtain a second-level prediction result corresponding to that second-level learner; and inputting the second-level prediction results output by all second-level learners into the linear regression model to obtain the representative sentiment value corresponding to the image sentiment prediction model.

[0013] According to an optional embodiment of the present invention, the step of predicting the sentiment value corresponding to the remaining sentiment words in the sentiment word set of the target image based on the representative sentiment value of each sentiment word set includes: for each sentiment word set, constructing a multiple linear regression model between the representative sentiment word of the sentiment word set and each remaining sentiment word of the sentiment word set using the least squares method; and predicting the sentiment value of the remaining sentiment words corresponding to the multiple linear regression model based on the representative sentiment value of the representative sentiment word of the sentiment word set and the multiple linear regression model.

[0014] Another aspect of the present invention provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, which, when executed by the processor, implement the method of the present invention.

[0015] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of the present invention.

[0016] Another aspect of the present invention provides a computer program including computer-executable instructions, which, when executed, are used to implement the methods of the embodiments of the present invention.

[0017] According to an embodiment of the present invention, multiple sentiment adjectives (i.e., sentiment words) are grouped based on correlation, and the prediction result of one sentiment adjective in each group is obtained using the W-stacking (weight allocation-stacking) model. Then, the prediction result of the remaining sentiment adjectives in each group is obtained by using the multiple linear regression method. This can at least partially solve the problem that the image sentiment prediction in related technologies cannot effectively predict the degree of emotional arousal of the image, and thus can achieve the technical effect of accurately predicting the degree of emotional arousal of the image.

[0018] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the invention. Attached Figure Description

[0019] The accompanying drawings, which are part of the specification of this invention, illustrate exemplary embodiments of the invention. The drawings, together with the description in the specification, serve to illustrate the principles of the invention.

[0020] Figure 1 This is a flowchart illustrating a fine-grained emotion prediction method for images, provided for a specific embodiment of the present invention.

[0021] Figure 2 This is a flowchart illustrating a specific embodiment of the present invention for obtaining a set of multiple emotion words.

[0022] Figure 3 This is a flowchart illustrating a process for predicting the representative sentiment value corresponding to each representative sentiment word in a target image, as provided in a specific embodiment of the present invention.

[0023] Figure 4 This is a schematic diagram illustrating a process for predicting the representative sentiment value corresponding to each representative sentiment word based on dimensionality reduction image features, as provided in a specific embodiment of the present invention.

[0024] Figure 5 This is a flowchart illustrating a process for establishing an image sentiment prediction model for each representative sentiment word based on an ensemble learning method, as provided in a specific embodiment of the present invention.

[0025] Figure 6 This is a flowchart illustrating a specific embodiment of the present invention for obtaining the representative sentiment value corresponding to a representative sentiment word.

[0026] Figure 7 This is a schematic diagram of the structure of an image sentiment prediction model provided for a specific embodiment of the present invention.

[0027] Figure 8 This is a flowchart illustrating the process of predicting the sentiment values ​​of the remaining sentiment words in a set of sentiment words for a target image, as provided in a specific embodiment of the present invention.

[0028] Figure 9 This is a block diagram of an electronic device for fine-grained emotion prediction of images, provided for a specific embodiment of the present invention.

[0029] Explanation of reference numerals in the attached figures:

[0030] Operations S101~S103 Operations S1011~S1013

[0031] Operations S1021~S1023 and S10231~S10232

[0032] Operations S102311~S102313 and S102321~S102323

[0033] S1031~S1032 Operation

[0034] 900 Electronic Devices 901 Processor

[0035] 902 ROM 903 RAM

[0036] 904 Bus 905 I / O Interface

[0037] 906 Input Section 907 Output Section

[0038] 908 Storage Section 909 Communication Section

[0039] 910 Driver 911 Removable Media Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the spirit of the contents disclosed in the present invention will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of the present invention, any person skilled in the art can make changes and modifications based on the technology taught in the present invention without departing from the spirit and scope of the present invention.

[0041] The illustrative embodiments and descriptions of the present invention are used to explain the invention, but are not intended to limit the invention. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0042] The terms "first," "second," etc., used in this document are not intended to specifically refer to order or sequence, nor are they intended to limit the invention. They are merely used to distinguish elements or operations described using the same technical terms.

[0043] The directional terms used in this article, such as up, down, left, right, front, or back, are for reference only when referring to the accompanying drawings. Therefore, the use of directional terms is for illustrative purposes and not to limit this work.

[0044] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0045] The term "and / or" as used herein includes any or all of the things mentioned.

[0046] The term "multiple" in this article includes "two" and "more than two"; the term "multiple groups" in this article includes "two groups" and "more than two groups".

[0047] The terms "approximately," "about," etc., used herein are intended to modify any quantity or error that may vary slightly, but these slight variations or errors do not change the essence of the quantity or error. Generally, the range of slight variations or errors modified by such terms may be 20% in some embodiments, 10% in others, 5% in still others, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0048] Certain terms used to describe this application will be discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art in describing the application.

[0049] Figure 1 This is a flowchart illustrating a fine-grained emotion prediction method for images, provided for a specific embodiment of the present invention.

[0050] In optional embodiments of the present invention, such as Figure 1 As shown, the image fine-grained sentiment prediction method may include the following operations S101 to S103:

[0051] In operation S101, multiple sentiment words describing the sentiment of an image are grouped based on the correlation between sentiment words to obtain multiple sentiment word sets, wherein each sentiment word set has a representative sentiment word.

[0052] In an optional embodiment of the present invention, the emotion words, also known as emotion adjectives, can include positive and negative emotion adjectives. For example, the present invention can use 18 emotion words, including 11 positive emotion words and 7 negative emotion words. The 11 positive emotion words can include: comfortable, atmospheric, happy, relaxed, dreamy, sunny, warm, romantic, hopeful, fresh, and cozy. The 7 negative emotion words can include: lonely, depressed, worried, repressed, fearful, sad, and lost. For each image, a 5-level scale can be used to represent the arousal level of each emotion word, where level 1 indicates no arousal of this type of emotion, level 2 indicates mild arousal, level 3 indicates perceptible arousal, level 4 indicates significant arousal, and level 5 indicates very strong arousal. Therefore, for each image, the arousal level of 18 emotion words (i.e., fine-grained emotional arousal perception level) can be obtained.

[0053] In an optional embodiment of the present invention, principal component analysis is used to obtain the factor loading matrix. Based on the factor loading matrix, multiple factors and multiple sets of emotional words are determined. These factors can explain more than 90% of the information in all emotional words, providing a relatively comprehensive explanation of all emotional words. For example, the nine emotional words "romantic," "warm," "cozy," "happy," "hopeful," "relaxed," "comfortable," "fresh," and "dreamy" are strongly correlated with the first factor; the five emotional words "loss," "depression," "loneliness," "worry," and "oppression" are strongly correlated with the second factor; "sadness" and "sunshine" are strongly correlated with the third factor; "fear" is strongly correlated with the fourth factor; and "atmosphere" is strongly correlated with the fifth factor. The emotional words with strong correlations can be grouped into a set. Therefore, five sets of emotional words can be obtained: the first set is "dreamy," "romantic," "warm," "cozy," "happy," "hopeful," "relaxed," "comfortable," and "fresh"; the second set is "oppression," "loss," "depression," "loneliness," and "worry"; the third set is "sunshine" and "sadness"; the fourth set is "fear"; and the fifth set is "atmosphere." We can then determine the representative sentiment word for each sentiment word set based on the prediction accuracy of each sentiment word. Therefore, we can determine that the representative sentiment word for the first sentiment word set is "dreamy," the representative sentiment word for the second is "oppressive," the representative sentiment word for the third is "sunshine," the representative sentiment word for the fourth is "fear," and the representative sentiment word for the fifth is "atmosphere." Thus, from the 18 sentiment words, the representative sentiment words are determined to be dreamy, oppressive, sunny, fearful, and atmospheric.

[0054] Next, in operation S102, the representative sentiment value corresponding to each representative sentiment word in the target image is predicted.

[0055] In an optional embodiment of the present invention, sentiment prediction can be performed directly using the initial image features of the target image to obtain the sentiment value corresponding to each representative sentiment word. Considering that directly using multi-dimensional initial image features for sentiment prediction may result in inaccurate predictions due to the redundancy of the initial image features, the initial image features can be dimensionality reduced, and the dimensionality-reduced image features can be used for sentiment prediction.

[0056] Then, in operation S103, the sentiment values ​​corresponding to the remaining sentiment words in the sentiment word set of the target image are predicted based on the representative sentiment values ​​of the representative sentiment words in each sentiment word set.

[0057] In an optional embodiment of the present invention, the sentiment value corresponding to the remaining sentiment words of the target image can be predicted based on the correlation between the representative sentiment words and the remaining sentiment words in each sentiment word set.

[0058] In an optional embodiment of the present invention, multiple emotional words are grouped based on the correlation between emotional words. The representative emotional value of the representative emotional word in each emotional word set (i.e., emotional word combination) is obtained using an image emotion prediction model (i.e., W-stacking model). The emotional value of the remaining emotional words in each emotional word set is obtained using a multiple linear regression method, thereby obtaining the emotional value corresponding to all emotional words. This conforms to the Chinese emotional cognition system, is applicable to the perception of richer emotional content, and can accurately predict the degree of arousal of different emotions in the target image.

[0059] The following is for reference. Figures 2-8 In conjunction with specific embodiments of the present invention, Figure 1 The method shown will be further explained.

[0060] Figure 2 This is a flowchart illustrating a specific embodiment of the present invention for obtaining a set of multiple emotion words.

[0061] In optional embodiments of the present invention, such as Figure 2 As shown, operation S101 groups multiple sentiment words describing the sentiment of an image based on the correlation between sentiment words to obtain multiple sentiment word sets, which may include the following operations S1011 to S1013:

[0062] In operation S1011, principal component analysis is used to obtain the factor loading matrix of multiple sentiment words, wherein the factor loading matrix is ​​composed of factor loading values.

[0063] In an optional embodiment of the present invention, principal component analysis is used to obtain the factor loading matrix, and then multiple factors and multiple sentiment word sets are determined based on the factor loading matrix. These factors can explain 92.498% of the information of 18 sentiment words, and can comprehensively explain the 18 sentiment words.

[0064] Then, in operation S1012, multiple factors and multiple sentiment word sets are determined based on the factor loading matrix, wherein the factor corresponding to the maximum absolute factor loading value of the multiple sentiment words corresponds one-to-one with the sentiment word set.

[0065] In an optional embodiment of the present invention, the factor loading matrix can be used to determine which sentiment words are correlated, thereby dividing multiple sentiment words into multiple sentiment word sets. The factor loading matrix consists of factor loading values, with each column corresponding to a factor. The factor loading values ​​reflect which factors can primarily explain the variation of each variable, i.e., they reflect the correlation coefficients between the 18 sentiment words and the factors. The factor corresponding to the largest absolute factor loading value of a sentiment word corresponds one-to-one with the sentiment word set. The larger the absolute value of the factor loading value, the closer the relationship between the 18 sentiment words and the factors corresponding to the factor loading value. The factor loading values ​​of the 18 sentiment words form the factor loading matrix. As shown in Table 1 below, the factor loading matrix consists of 5 columns of factor loading values, corresponding to 5 factors. The largest absolute factor loading value for "romantic" is 0.862 (i.e., the largest absolute factor loading value), corresponding to factor 1; the largest absolute factor loading value for "warm" is 0.850, corresponding to factor 1; the largest absolute factor loading value for "cozy" is 0.810, corresponding to factor 1; the largest absolute factor loading value for "happy" is 0.723, corresponding to factor 1; the largest absolute factor loading value for "hopeful" is 0.738, corresponding to factor 1; the largest absolute factor loading value for "relaxed" is 0.732, corresponding to factor 1; the largest absolute factor loading value for "comfortable" is 0.739, corresponding to factor 1; the largest absolute factor loading value for "fresh" is 0.718, corresponding to factor 1; and the largest absolute factor loading value for "dreamy" is 0.663, corresponding to factor 1. It can be seen that the nine emotional words "romantic," "warm," "cozy," "happy," "hopeful," "relaxed," "comfortable," "fresh," and "dreamy" have a strong correlation with factor 1. Therefore, it can be said that these nine emotional words have a strong correlation. The largest absolute factor loading for "loss" is 0.820, corresponding to factor 2; the largest absolute factor loading for "depression" is 0.780, corresponding to factor 2; the largest absolute factor loading for "loneliness" is 0.859, corresponding to factor 2; the largest absolute factor loading for "anxiety" is 0.683, corresponding to factor 2; and the largest absolute factor loading for "repression" is 0.600, corresponding to factor 2. This shows that the five emotion terms "loss," "depression," "loneliness," "anxiety," and "repression" have a strong correlation with factor 2. Therefore, there is a strong correlation among these five emotion terms. The largest absolute factor loading for "sadness" is 0.787, corresponding to factor 4; and the largest absolute factor loading for "sunshine" is 0.821, corresponding to factor 4. The two emotion terms "sadness" and "sunshine" have a strong correlation with factor 4. Therefore, there is a strong correlation between these two emotion terms. The largest absolute factor loading for fear is 0.765, corresponding to factor 3. Therefore, fear is strongly correlated with factor 3. There are no sentiment words that are strongly correlated with fear.The maximum absolute factor loading value for the atmosphere is 0.944, corresponding to factor 5. Therefore, the atmosphere has a strong correlation with factor 5, and there are no sentiment words with a strong correlation to the atmosphere. Therefore, it can be determined that the first set of sentiment words is dreamy, romantic, warm, cozy, happy, hopeful, relaxed, comfortable, and refreshing, which can be well explained by factor 1; the second set of sentiment words is depressed, lost, dejected, lonely, and anxious, which can be well explained by factor 2; the third set of sentiment words is sunshine and sadness, which can be well explained by factor 4; the fourth set of sentiment words is fear, which can be well explained by factor 3; and the fifth set of sentiment words is the atmosphere, which can be well explained by factor 5.

[0066] Table 1

[0067]

[0068] Next, in operation S1013, the representative sentiment word for each sentiment word set is determined.

[0069] In an optional embodiment of the present invention, operation S1013, determining the representative sentiment word for each sentiment word set, may include the following operations: constructing an initial image sentiment prediction model for each sentiment word using a random forest algorithm; determining the prediction accuracy of the sentiment words within each sentiment word set based on the initial image sentiment prediction model; and determining the representative sentiment word for each sentiment word set based on the prediction accuracy. Ultimately, the words "dreamy," "oppressive," "sunshine," "fear," and "atmosphere" are determined to have the highest prediction accuracy in their respective sentiment word sets. Therefore, the representative sentiment word for the first sentiment word set is "dreamy," the representative sentiment word for the second sentiment word set is "oppressive," the representative sentiment word for the third sentiment word set is "sunshine," the representative sentiment word for the fourth sentiment word set is "fear," and the representative sentiment word for the fifth sentiment word set is "atmosphere."

[0070] In an optional embodiment of the present invention, operation S1013, which determines the representative sentiment word of each sentiment word set, may further include the following operations: using the Least Absolute Shrinkage Selection Operator (LASSO) and importance dimensionality reduction to filter the sentiment word with the most features in each sentiment word set; and determining the sentiment word with the most features as the representative sentiment word of the corresponding sentiment word set.

[0071] In an optional embodiment of the present invention, directly using the image sentiment prediction model corresponding to each sentiment word to predict the sentiment value corresponding to all sentiment words results in an excessively large overall computational load, high requirements for device configuration, and low prediction efficiency. If a single image sentiment prediction model is used to predict the sentiment values ​​corresponding to all sentiment words simultaneously, the prediction accuracy cannot be guaranteed. Therefore, for the representative sentiment word in each sentiment word set, the image sentiment prediction model corresponding to each representative sentiment word is directly used to predict the representative sentiment value corresponding to the representative sentiment word; for the remaining sentiment words in each sentiment word set, a multiple linear regression model between the representative sentiment word and the remaining sentiment words is used to predict the sentiment values ​​of the remaining sentiment words. This reduces device configuration and data computation while ensuring prediction accuracy.

[0072] Figure 3 This is a flowchart illustrating a process for predicting the representative sentiment value corresponding to each representative sentiment word in a target image, as provided in a specific embodiment of the present invention.

[0073] In optional embodiments of the present invention, such as Figure 3 As shown, operation S102 predicts the representative sentiment value corresponding to each representative sentiment word in the target image, which may include the following operations S1021 to S1023:

[0074] In operation S1021, multidimensional initial image features of the target image are extracted.

[0075] In an optional embodiment of the present invention, in order to improve the performance of the image sentiment prediction model, the present invention combines relevant theories of art, psychology, and photography to extract low-level features, high-level features, and key region features that conform to the characteristics of scene image expression, totaling 250 dimensions. The extraction methods for each dimension are as follows:

[0076] The underlying features can include color features and texture features. Color moments and color contrast are selected to describe the color features of the image. An image can be described using 160-dimensional underlying color feature vectors S1 to S160. The gray-level co-occurrence matrix (GLCM) and Tamura texture features of the image are extracted to describe the texture information of the scene image. In the GLCM feature extraction process, based on the meaning of each feature and experimental results, this invention selects five features that can represent the image texture: energy (second-order moment), entropy, moment of inertia (contrast), correlation, and uniformity, forming a 20-dimensional texture feature vector, denoted as G1 to G20. Tamura texture features are proposed based on psychological research theories about people's visual perception of texture. This invention selects to extract three-dimensional Tamura texture features of image roughness, contrast, and orientation, denoted as T1 to T3, based on the meaning of each feature.

[0077] High-level features can include aesthetic and compositional features. These features can describe semantic information more strongly correlated with emotion, and to some extent, can bridge the "semantic gap" caused by research based solely on low-level features. This invention extracts aesthetic and compositional features to describe the high-level features of scene images. Aesthetic features include texture complexity (M), color complexity (C1-C6), energy features (E1-E12), diversity features (V1-V11), gradation features (G1-G5), and motion features (M1-M10). Compositional features include the rule of thirds (RT1-RT3), depth of field features (LD1-LD3), and dynamic features (D1-D6).

[0078] Key region features include color moment features and shape contrast features. Different regions of the same image can evoke different emotional categories and intensities in viewers. During image viewing, viewers often focus their attention on areas of interest rather than the entire image. Therefore, extracting features from key regions of an image is extremely important. This invention uses the IG algorithm to calculate the saliency of the image and performs region segmentation based on the mean-shift algorithm to obtain key regions. Based on this, color moment features and shape contrast features are extracted from the key regions. The color moment features are the first, second, and third moments of the image in the three channels of the HSV color space, and the shape contrast features are the proportion of the total number of pixels in the key regions to the total number of pixels in the original image. Ultimately, this invention obtains 10-dimensional key region features of a scene image, including 9-dimensional color moment features and 1-dimensional shape contrast features, denoted as R1 to R10.

[0079] Next, in operation S1022, for each sentiment word, the multidimensional initial image features are reduced in dimensionality using a combination of minimum absolute shrinkage selection operator and random forest feature importance reduction method to obtain dimensionality-reduced image features.

[0080] In an optional embodiment of the present invention, considering that directly using multi-dimensional initial image features for sentiment prediction may lead to inaccurate prediction results due to the redundancy of the initial image features, the initial image features can be dimensionality reduced, and the dimensionality-reduced image features can be used for sentiment prediction. For each sentiment word, the multi-dimensional initial image features are dimensionality-reduced using a combination of Least Absolute Shrinkage and Selection Operator (LASSO) and Random Forest (RF) feature importance dimensionality reduction method to obtain dimensionality-reduced image features. The representative features extracted based on the characteristics of the scene image cover more sentiment distribution factors, can more comprehensively describe the sentiment information of the image, and improve prediction accuracy.

[0081] Taking the sentiment word "dreamy" as an example, after LASSO regression analysis, the initial 250-dimensional image features were reduced to 61-dimensional image features. Based on this, random forest importance analysis further reduced the 61-dimensional image features to 53-dimensional image features. Similarly, taking the sentiment word "sad" as an example, after LASSO regression analysis, the initial 250-dimensional image features were reduced to 8-dimensional image features. Based on this, random forest importance analysis further reduced the 8-dimensional image features to 7-dimensional image features. Through these operations, 18 sets of dimensionality-reduced image features corresponding to the sentiment words were obtained, each containing a different number of features, as shown in Table 2 below.

[0082] Table 2

[0083]

[0084]

[0085] Then, in operation S1023, the representative sentiment value corresponding to each representative sentiment word is predicted based on the dimensionality reduction image features.

[0086] In an optional embodiment of the present invention, an image sentiment prediction model corresponding to each representative sentiment word is constructed using an ensemble learning method, and the representative sentiment value corresponding to each representative sentiment word is obtained using the image sentiment prediction model based on the dimensionality-reduced image features.

[0087] In an optional embodiment of the present invention, the representative sentiment value corresponding to each representative sentiment word is predicted using dimensionality-reduced image features, which can avoid inaccurate prediction results caused by image feature redundancy and improve the accuracy of prediction results.

[0088] Figure 4 This is a schematic diagram illustrating a process for predicting the representative sentiment value corresponding to each representative sentiment word based on dimensionality reduction image features, as provided in a specific embodiment of the present invention.

[0089] In optional embodiments of the present invention, such as Figure 4 As shown, operation S1023 predicts the representative sentiment value corresponding to each representative sentiment word based on the dimensionality reduction image features, and may include, for example, the following operations S10231 to S10232:

[0090] In operation S10231, an image sentiment prediction model corresponding to each representative sentiment word is established based on the ensemble learning method.

[0091] In an optional embodiment of the present invention, ensemble learning is a popular machine learning strategy that trains multiple learners and combines their different outputs to solve a problem. An ensemble algorithm can be composed of multiple learners and has stronger generalization ability than a single learner, transforming weak learners into strong learners capable of accurate prediction. For example, the random forest algorithm can be viewed as a single weak learner used to compare the initial sentiment values ​​of various sentiment words. The image sentiment prediction model constructed in this operation can be viewed as a strong learner capable of accurately predicting the sentiment values ​​corresponding to sentiment words.

[0092] Next, in operation S10232, the dimensionality-reduced image features are input into the image sentiment prediction model corresponding to each representative sentiment word to obtain the representative sentiment value corresponding to that representative sentiment word.

[0093] In an optional embodiment of the present invention, each representative sentiment word corresponds to an image sentiment prediction model. By inputting the dimensionality-reduced image features into the image sentiment prediction model corresponding to the representative sentiment word, the representative sentiment value corresponding to the representative sentiment word can be obtained.

[0094] In an optional embodiment of the present invention, an ensemble learning method is used to train an image sentiment prediction model corresponding to each representative sentiment word, which can accurately predict the representative sentiment value corresponding to the representative sentiment word.

[0095] Figure 5 This is a flowchart illustrating a process for establishing an image sentiment prediction model for each representative sentiment word based on an ensemble learning method, as provided in a specific embodiment of the present invention.

[0096] In optional embodiments of the present invention, such as Figure 5 As shown, operation S10231 establishes an image sentiment prediction model corresponding to each representative sentiment word based on the ensemble learning method, and may include the following operations S102311 to S102313:

[0097] In operation S102311, a scene image library containing multiple images and covering all original emotions is constructed.

[0098] In an optional embodiment of this invention, the standards and principles for establishing IAPS (International Emotional Image System) and CAPS (Chinese Emotional Image System) can be referenced. Images of various film and television scenes were selected using film review websites such as Douban, and balanced from multiple aspects including color, people, animals, architecture, and natural scenery. Ultimately, a scene image library containing 700 images covering all emotions was established. All scene images were uniformly processed to a resolution of 1280×720 and in JPG format.

[0099] Next, in operation S102312, most of the images in the scene image library are used as the initial accuracy training set, and the remaining images in the scene image library are used as the initial accuracy test set.

[0100] In an optional embodiment of the present invention, the image sentiment prediction model may include, for example, four first-level learners and three second-level learners. 325 scene images out of 700 scene images can be used as the training set for the first-level learners to train four first-level learners; 325 scene images out of 700 scene images can be used as the training set for the second-level learners to train three second-level learners and one linear regression model; the remaining 50 scene images can be used as the overall test set.

[0101] Then, in operation S102313, the image sentiment prediction model corresponding to each representative sentiment word is trained using the initial accuracy training set and the initial accuracy test set, thereby constructing the image sentiment prediction model corresponding to each representative sentiment word.

[0102] In an optional embodiment of the present invention, the outputs of the four first-level learners are used as the inputs of each second-level learner. The first-level learners, second-level learners, and linear regression models are trained using an initial accuracy training set, and the image sentiment prediction model is tested using an initial accuracy test set.

[0103] In an optional embodiment of the present invention, a standard scene image library is constructed, and the image sentiment prediction model corresponding to each representative sentiment word is trained and tested using the images in the scene image library. The image sentiment prediction model corresponding to each representative sentiment word can be used to accurately predict the representative sentiment value corresponding to each representative sentiment word, thereby achieving fine-grained image sentiment prediction.

[0104] Figure 6 This is a flowchart illustrating a specific embodiment of the present invention for obtaining the representative sentiment value corresponding to a representative sentiment word.

[0105] In optional embodiments of the present invention, such as Figure 6 As shown, operation S10232 inputs the dimensionality-reduced image features into the image sentiment prediction model corresponding to each representative sentiment word to obtain the representative sentiment value corresponding to that representative sentiment word, and may include the following operations S102321 to S102323:

[0106] In operation S102321, the dimensionality-reduced image features are input into each first-level learner of the image sentiment prediction model to obtain the first-level prediction result corresponding to that first-level learner. Each image sentiment prediction model includes multiple first-level learners, multiple second-level learners, and a linear regression model.

[0107] In an optional embodiment of the present invention, each image sentiment prediction model includes four first-level learners, three second-level learners, and one linear regression model. A schematic diagram of the image sentiment prediction model is shown below. Figure 7 As shown, the four first-level learners can employ SVR (Support Vector Regression), RF (Random Forest), Catboost (Gradient Boosting + Categorical Features), and MLP (Multilayer Perceptron) algorithms, respectively. Catboost is an improvement on the GBDT (Gradient Boosting Tree) algorithm. Compared with other boosting algorithms, Catboost can effectively avoid overfitting and efficiently handle categorical variables. During modeling, no preprocessing of categorical variables is required; Catboost can directly convert categorical variables into numerical variables. Simultaneously, Catboost can combine original image features based on the relationships between them, enriching the feature dimensions to a certain extent and further improving prediction accuracy. The second-level learners can employ SVR, RF, and Catboost algorithms, respectively. The dimensionality-reduced image features are input into each first-level learner, and each first-level learner outputs a first-level prediction result.

[0108] Next, in operation S102322, the first-level prediction results output by all first-level learners are input into each second-level learner to obtain the second-level prediction result corresponding to that second-level learner.

[0109] In an optional embodiment of the present invention, the first-level prediction results output by the four first-level learners are respectively input to each second-level learner as input to the second-level learner. Weights are assigned to each second-level learner using the least squares method. Each second-level learner outputs a second-level prediction result.

[0110] Then, in operation S102323, the secondary prediction results output by all secondary learners are input into the linear regression model to obtain the representative sentiment value corresponding to the image sentiment prediction model.

[0111] In an optional embodiment of the present invention, the secondary prediction results output by three secondary learners are input into a linear regression model to obtain the representative sentiment value of the representative sentiment word. The mathematical expression of the linear regression model can be: Among them, a i denoted by , b represents the weight (regression coefficient); b represents the constant term; y represents the secondary prediction result; Y represents the sentiment value; i takes values ​​of 1, 2, and 3. The mathematical expressions for the linear regression models corresponding to the sentiment words dreamy, repressed, fearful, sunny, and atmospheric are shown in Table 3 below.

[0112] Table 3

[0113] Representative emotional words Mathematical expression of linear regression model <![CDATA[R 2 <!-- 10 -->]]> dream <![CDATA[E 梦幻 =1.032R SVR -0.616R RF +1.150R Catboost -0.176]]> 0.92 Suppression <![CDATA[E 压抑 =0.555R SVR -0.560R RF +1.738R Catboost -0.254]]> 0.90 fear <![CDATA[E 恐惧 =1.325R SVR -0.634R RF +1.610R Catboost -0.251]]> 0.89 Sunlight <![CDATA[E 阳光 =1.389R SVR -1.418R RF +1.751R Catboost -0.282]]> 0.92 atmosphere <![CDATA[E 大气 =1.104R SVR -1.052R RF +1.633R Catboost -0.263]]> 0.88

[0114] In Table 3 above, E 梦幻 E 压抑 E 恐惧 E 阳光 E 大气 These are the representative emotional values ​​for the emotional words "dreamy," "oppressive," "fearful," "sunny," and "magnificent." R SVR ,R RF ,R Catboost The outputs of the second-level learners represent the results of the SVR, RF, and Catboost algorithms, respectively. R 2 R is used to measure the goodness of fit of a linear regression model, and its value is usually between 0 and 1. 2 The closer the value is to 1, the better the linear regression model fits. In this invention, the linear regression model showed a high degree of fit for the prediction results of the five representative sentiment words.

[0115] In an optional embodiment of the present invention, the present invention improves the relevant stacking algorithm by adjusting the number and ensemble method of the second-level learners, and assigns weights to each learner using the least squares method. The improved algorithm is denoted as the W-stacking (weight allocation-stacking) ensemble learning model, which is used to predict the representative sentiment values ​​of five representative sentiment words: "dreamy, oppressive, sunny, fearful, and atmospheric," achieving effective prediction of fine-grained sentiment in target images. The W-stacking ensemble learning model yields an average MAE (mean absolute error) of 0.12 and an average MSE (mean squared error) of 0.03 for the representative sentiment values ​​of the five representative sentiment words. This is superior to stacking algorithms using only one second-level learner, superior to stacking methods with multiple second-level learners with fixed weights, and also superior to single machine learning algorithms such as decision trees, Adaboost, SVR, RF, Catboost, and MLP. The W-stacking ensemble learning model based on linear regression coefficients optimizes the prediction performance of each first-level learner, making the prediction results more accurate and robust.

[0116] Figure 8 This is a flowchart illustrating the process of predicting the sentiment values ​​of the remaining sentiment words in a set of sentiment words for a target image, as provided in a specific embodiment of the present invention.

[0117] In optional embodiments of the present invention, such as Figure 8 As shown, operation S103 predicts the sentiment value corresponding to the remaining sentiment words in the sentiment word set of the target image based on the representative sentiment value of the representative sentiment word in each sentiment word set, and may include the following operations S1031 to S1032:

[0118] In operation S1031, for each sentiment word set, a multiple linear regression model is constructed using the least squares method between the representative sentiment word of the sentiment word set and each of the remaining sentiment words in the sentiment word set.

[0119] In an optional embodiment of the present invention, multiple linear regression is a statistical method for studying the relationship between a dependent variable and multiple independent variables. Assuming the dependent variable is represented by Y and the independent variables by X, when the number of independent variables is p, X = [x1, x2, ... x...]. p The goal is to find a set of β = [β1, β2, ... β] using some method. p The parameter estimation of ] makes The condition is met, where ε follows a normal distribution with a mean of 0. The method for finding the parameter estimate is the least squares method. In this invention, the sentiment value of each remaining sentiment word to be predicted is considered as the dependent variable, and the number of features of its corresponding dimensionality-reduced image features (see Table 2) and the representative sentiment values ​​of representative sentiment words that have an intrinsic correlation with the remaining sentiment word are considered as independent variables. A multiple linear regression model between the dependent and independent variables is established using the least squares method.

[0120] In an optional embodiment of the present invention, taking the sentiment word "sadness" as an example, according to the analysis above, "sadness" and "sunshine" belong to the same set of sentiment words, so prediction can be made using the representative sentiment value of the sentiment word "sunshine". Meanwhile, the dimensionality-reduced image features corresponding to the remaining sentiment word "sadness" contain a total of 8 feature quantities, denoted as S3 (representing the 3rd dimension color feature), S... 39 (representing the 39th dimension of color feature), S 63 (representing the 63rd dimension of color feature), S 90 The following are used to represent the 90th dimension: color (RT3), composition (RT3), color complexity (C1), color complexity (C3), and diversity (V4). In constructing the linear regression model, to maximize the representation of the correlation between different emotions and enhance the model's fitting performance, the emotion values ​​(non-predicted emotion values) of 650 images from the scene image library (excluding the initial accuracy test set) can be used to build the model, i.e., the following equation can be established:

[0121] Y' 伤感 =β0+β1Y' 阳光 +β2S3+β3S 39 +β4S 63 +β5S 90 +β6RT3+β7C1+β8C3+β9V4

[0122] In the formula, Y' 阳光 Y' represents the truth value of the sentiment word "sunshine";伤感 The linear regression result represents the sentiment value of the remaining sentiment word "sad"; β0, β1, ..., β9 represent the regression coefficients of each independent variable. The method for predicting the sentiment value of "sad" using this model is as follows: For each test image in the initial accuracy test set, we have:

[0123] Y 伤感 =β0+β1X 阳光 +β2S3+β3S 39 +β4S 63 +β5S 90 +β6RT3+β7C1+β8C3+β9V4

[0124] In the formula, X 阳光 Y represents the sentiment value of the sentiment word "sunshine" obtained through the W-stacking model; 伤感 This represents the final sentiment value obtained by predicting the remaining sentiment word "sadness".

[0125] Then, in operation S1032, the sentiment values ​​of the remaining sentiment words corresponding to the multiple linear regression model are predicted based on the representative sentiment values ​​of the representative sentiment words in the sentiment word set and the multiple linear regression model.

[0126] In an optional embodiment of the present invention, for each remaining sentiment word that relies on sentiment relevance for sentiment prediction, the number of independent variables in the linear regression model built upon this model varies due to the different number of features in the dimensionality-reduced image features. For example, the remaining sentiment word "romantic" has 40 independent variables, while the remaining sentiment word "warm" has 34. These independent variables still have a high dimensionality, which is detrimental to the prediction of the linear regression model. Therefore, the initial linear regression model built upon the feature count of these dimensionality-reduced image features is further optimized. Specifically, the "stepwise regression" method in linear regression is used to screen the independent variables. When the significance value of a single independent variable exceeds a certain standard, it is removed. The final retained variables satisfy two conditions: 1) each independent variable contributes significantly to the dependent variable; 2) when the linear regression model contains the above independent variables, its significance probability value is much less than 0.01. Based on the above method, the initial linear regression models for the 13 emotional words "romantic, warm, cozy, comfortable, hopeful, relaxed, happy, refreshing, lonely, lost, depressed, worried, and sad" were optimized through stepwise regression to obtain simplified linear regression models.

[0127] In an optional embodiment of the present invention, based on the known representative emotional value of the emotional word "dreamy," the multiple linear regression models corresponding to the remaining emotional words in the same set of emotional words—"romantic," "warm," "comfortable," "hopeful," "relaxed," "happy," and "fresh"—and "dreamy" are shown in Table 4 below. In the table, d1, d2, ... d8 represent the regression equation coefficients of the representative emotional value of "dreamy" in each linear regression model; e1, e2, ... e8 represent the regression equation coefficients of the color complexity feature C1 in each linear regression model; g1, g2, ... g8 represent the constant terms of each linear regression model. Let S represent the coefficients of the color feature vector S in each linear regression model, where m∈{2,3,4,5,6,7,8} and n∈{18,21,45,57,63,81,88,97,111,120,129}. For example... h1 represents the regression coefficient of the color feature vector S45 in the linear regression prediction model for "warmth"; h2 represents the regression coefficient of the color complexity C2 in the emotional perception prediction model for "comfort".

[0128] Table 4

[0129]

[0130]

[0131] Table 5 below shows examples of multiple linear regression models corresponding to the remaining sentiment words "romance, warmth, comfort, hope, relaxation, happiness, and freshness" based on the known sentiment value of the sentiment word "dreamy".

[0132] Table 5

[0133] Remaining emotional words Linear regression model romantic <![CDATA[Y 浪漫 =0.878X 梦幻 +0.287C1-0.160]]> Warm <![CDATA[Y 温馨 =0.652X 梦幻 +0.406C1+0.277S 45 +0.144S 88 -0.255]]> warmth <![CDATA[Y 温暖 =0.633X 梦幻 +0.494*C1+0.264S 63 +0.193S 81 -0.288]]> Comfortable <![CDATA[Y 舒服 =0.695X 梦幻 +0.165C1+0.182S 21 +0.169S 120 +0.246C2-0.177]]> hope <![CDATA[Y 希望 =0.658X 梦幻 +0.333C1+0.179S 21 +0.153S 81 -0.160]]> easy <![CDATA[Y 轻松 =0.660X 梦幻 +0.385C1+0.191S 57 +0.174S 45 -0.186]]> hapiness <![CDATA[Y 快乐 =0.535X 梦幻 +0.323C1+0.188S 18 +0.189S 129 +0.125S 97 -0.217]]> Fresh <![CDATA[Y 清新 =0.650X 梦幻 +0.175S 111 +0.320C1+0.168S 57 -0.169]]>

[0134] In an optional embodiment of the present invention, based on the known sentiment prediction results of the representative sentiment word "depression," the remaining sentiment words "loneliness, loss, depression, and anxiety," calculated according to relevance, and their corresponding multiple linear regression models with the representative sentiment word "depression" are shown in Table 6 below. In Table 6, d9, d 10 ,d 11 ,d 12 This represents the regression coefficients of the emotional value representing the sentiment word "suppression" in various linear regression models; g9,g 10 ,g 11 ,g 12 Represents the constant term in each linear regression equation; h1 represents the regression coefficient of the color feature vector S83 in the linear regression equation for "loss"; h2 represents the regression coefficient of the color complexity feature C2 in the linear regression equation for the remaining emotion word "loneliness".

[0135] Table 6

[0136]

[0137] In an optional embodiment of the present invention, Table 7 below shows the examples of multiple linear regression models corresponding to the remaining emotional words "loneliness, loss, depression and anxiety" and "repression" based on the known representative emotional value of the emotional word "repression".

[0138] Table 7

[0139] Remaining emotional words Linear regression model Lonely <![CDATA[Y 孤独 =0.684X 压抑 -0.180C2+0.301]]> Loss <![CDATA[Y 失落 =0.898X 压抑 -0.092S 83 +0.064]]> depressed <![CDATA[Y 消沉 =0.887X 压抑 +0.029]]> concern <![CDATA[Y 忧虑 =0.979X 压抑 +0.037]]>

[0140] In an optional embodiment of the invention, based on the known representative sentiment value of the sentiment word "sunshine," the linear regression model between the remaining sentiment word "sadness" and the representative sentiment word "sunshine," calculated according to the correlation between sentiment words, is shown in the following formula. In the formula, d 13 This represents the regression coefficient of the sentiment value of the emotional word "sunshine" in a linear regression model; g 13 This represents the constant term in the linear regression equation; h1 represents the regression coefficient of the color feature vector S109; h3 represents the regression coefficient of the energy feature E1.

[0141]

[0142] The following is an example of a linear regression model between the remaining sentiment word "sadness" and the representative sentiment word "sunshine" based on correlation calculation:

[0143] Y 伤感 =-0.750X 阳光 -0.359E1-0.112S 109 +0.771

[0144] Overall, the W-stacking + multiple linear regression image fine-grained sentiment prediction model employed in this invention effectively utilizes image visual features and the correlation characteristics between sentiment words, achieving high prediction accuracy while reducing model representation complexity. Test results on the same small sample dataset show that the proposed method outperforms fine-grained sentiment prediction results using a single machine learning model or partial deep transfer learning models (such as VGG16+SVR, Inception+SVR, ResNet50+SVR, etc.) in terms of the average sentiment value across 18 sentiment categories. The linear regression model combined with the correlation between sentiment words improves the prediction accuracy of sentiment words that are not easily observed directly through image features, making it suitable for fine-grained sentiment prediction and analysis on small sample data.

[0145] Figure 9 This is a block diagram of an electronic device for fine-grained emotion prediction of images, provided for a specific embodiment of the present invention. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0146] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0147] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 902 and / or RAM 903. It should be noted that programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0148] According to an embodiment of the present invention, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0149] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by processor 901, it performs the functions defined in the electronic device of the embodiments of the present invention. According to embodiments of the present invention, the electronic devices, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0150] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0151] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0154] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A fine-grained image sentiment prediction method, characterized in that, The method includes: Based on the correlation between sentiment words, multiple sentiment words describing the sentiment of an image are grouped to obtain multiple sentiment word sets, where each sentiment word set has a representative sentiment word; Predict the sentiment value corresponding to each sentiment word in the target image; and Based on the representative sentiment value of the representative sentiment word in each sentiment word set, predict the sentiment value corresponding to the remaining sentiment words in the sentiment word set of the target image. The step of predicting the sentiment values ​​corresponding to the remaining sentiment words in the sentiment word set of the target image based on the representative sentiment values ​​of the representative sentiment words of each sentiment word set includes: For each sentiment word set, a multiple linear regression model is constructed using the least squares method between the representative sentiment word of that set and each of the remaining sentiment words in that set; and Based on the representative sentiment values ​​of the representative sentiment words in the sentiment word set and the multiple linear regression model, predict the sentiment values ​​of the remaining sentiment words corresponding to the multiple linear regression model.

2. The image fine-grained sentiment prediction method according to claim 1, characterized in that, The steps of grouping multiple sentiment words describing image sentiment based on the correlation between sentiment words to obtain multiple sentiment word sets include: Principal component analysis was used to obtain factor loading matrices for multiple sentiment words, wherein the factor loading matrices are composed of factor loading values; Based on the factor loading matrix, multiple factors and multiple sets of sentiment words are determined, wherein the factor corresponding to the maximum absolute factor loading value of each sentiment word corresponds one-to-one with the sentiment word set; and Determine the representative sentiment word for each sentiment word set.

3. The image fine-grained sentiment prediction method according to claim 2, characterized in that, The steps to determine the representative sentiment word for each sentiment word set include: The random forest algorithm is used to construct an initial image sentiment prediction model for each sentiment word; The prediction accuracy of sentiment words within each sentiment word set is determined based on the initial image sentiment prediction model; and Based on the prediction accuracy, the representative sentiment word for each sentiment word set is determined.

4. The image fine-grained sentiment prediction method according to claim 1, characterized in that, The steps for predicting the sentiment value corresponding to each sentiment word in a target image include: Extract the multidimensional initial image features of the target image; For each sentiment word, dimensionality reduction is performed on the multidimensional initial image features using a combination of minimum absolute shrinkage selection operator and random forest feature importance to obtain dimensionality-reduced image features; and Predict the representative sentiment value corresponding to each representative sentiment word based on dimensionality reduction image features.

5. The image fine-grained sentiment prediction method according to claim 4, characterized in that, The steps for predicting the representative sentiment value for each representative sentiment word based on dimensionality-reduced image features include: An image sentiment prediction model corresponding to each representative sentiment word is established based on an ensemble learning method; and The reduced image features are input into the image sentiment prediction model corresponding to each sentiment word to obtain the sentiment value corresponding to that sentiment word.

6. The image fine-grained sentiment prediction method according to claim 5, characterized in that, The steps for building an image sentiment prediction model for each representative sentiment word based on ensemble learning methods include: Construct a scene image library containing multiple images and covering all original emotions; The majority of images in the scene image library are used as the initial accuracy training set, and the remaining images in the scene image library are used as the initial accuracy test set; and The image sentiment prediction model corresponding to each representative sentiment word is trained using the initial accuracy training set and the initial accuracy test set, thereby constructing the image sentiment prediction model corresponding to each representative sentiment word.

7. The image fine-grained sentiment prediction method according to claim 5, characterized in that, The steps of inputting the dimensionality-reduced image features into the image sentiment prediction model corresponding to each sentiment word to obtain the sentiment value corresponding to that sentiment word include: The dimensionality-reduced image features are input into each first-level learner of the image sentiment prediction model to obtain the first-level prediction result corresponding to that first-level learner. Each image sentiment prediction model includes multiple first-level learners, multiple second-level learners, and a linear regression model. Input the first-level prediction results output by all first-level learners into each second-level learner to obtain the second-level prediction result corresponding to that second-level learner; and The secondary prediction results output by all secondary learners are input into the linear regression model to obtain the representative sentiment value corresponding to the image sentiment prediction model.

8. An electronic device, comprising: One or more processors; A storage device for storing executable instructions, which, when executed by the processor, implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, implement the method according to any one of claims 1 to 7.

Images