Image sentiment analysis method and device, storage medium and electronic device

Abstract

The present invention provides an image sentiment analysis method and device, a storage medium and an electronic device, belonging to the field of image analysis, the method comprising: determining a metadata set of a target image, including descriptive text information and label information; converting the label information into a semantic description form to obtain label description information; uniformly representing the target image, descriptive text information and label description information to obtain metadata features and image features; performing information enhancement on the metadata features and image features to obtain metadata enhancement features and image enhancement features; performing cross-modal feature fusion processing on the metadata enhancement features and image enhancement features to obtain fusion features of the target image; performing emotion recognition based on the fusion features to obtain the emotion category of the target image. By applying the method of the present invention, the metadata features and visual features of the image can be used to perform multi-dimensional analysis on the emotions presented by the image, which is conducive to improving the accuracy of image emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of image analysis technology, and in particular to an image sentiment analysis method and device, a storage medium and an electronic device. Background Art

[0002] Data mining technology has been widely used in many fields to analyze data to meet business needs. Image sentiment analysis is one of the common tasks in data analysis scenarios. Image sentiment analysis aims to predict the emotional polarity expressed in an image based on a given image.

[0003] At present, the implementation method of image sentiment analysis is mainly to extract and analyze the visual features of the image, such as image color, facial expressions, body movements, etc., and use the visual features of the image to identify the emotions expressed in the image.

[0004] The inventors have discovered through research that in actual application scenarios, visual information in images is often easily disturbed by factors such as light, occlusion, and perspective, as well as factors such as facial expressions or body movements of characters. The same visual presentation may express different emotions in different scenarios. Therefore, the visual features extracted from the image may differ from the actual image representation or emotional expression. In existing image emotion analysis methods, the emotion recognition results of the image only depend on the visual features in the image. When there is a difference between the visual features of the image and the actual expression of the image, it will lead to large errors in the emotion analysis results, resulting in poor accuracy in emotion recognition. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides an image sentiment analysis method to solve the problem that in the existing image sentiment analysis method, when the visual features of the image are disturbed, the error of the sentiment analysis result is large and the accuracy of emotion recognition is low.

[0006] The embodiment of the present invention also provides an image sentiment analysis device to ensure the practical implementation and application of the above method.

[0007] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0008] An image sentiment analysis method, comprising:

[0009] When it is necessary to perform sentiment analysis on a target image, a metadata set of the target image is determined; the metadata set includes descriptive text information and tag information corresponding to the target image; the tag information includes tag data of multiple tag types;

[0010] Performing semantic description conversion processing on the tag information to obtain tag description information;

[0011] Performing unified characterization processing on the target image, the description text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the description text information, and label description features corresponding to the label description information;

[0012] The text description feature and the tag description feature are used as metadata features, and information enhancement processing is performed on the metadata features and the image features to obtain metadata enhancement features corresponding to the metadata features and image enhancement features corresponding to the image features;

[0013] Performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a fusion feature of the target image;

[0014] Emotion recognition is performed on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image.

[0015] In the above method, optionally, each of the tag types includes an object tag and a scene tag, and determining the metadata set of the target image includes:

[0016] Reading metadata information of the target image;

[0017] Determining whether the metadata information contains target attribute data; the target attribute data includes original image description text, original object label data and original scene label data;

[0018] If the metadata information does not contain the target attribute data, a preset image description model is applied to perform image description processing on the target image, and an image description text output by the image description model is obtained, and the image description text is used as the description text information corresponding to the target image; the image description model is a model obtained by training a BLIP model;

[0019] Applying a preset object recognition model to perform object recognition processing on the target image, obtaining each object identifier output by the object recognition model, and using each object identifier as label data of the object label; the object recognition model is a model obtained by training a fast regional convolutional neural network;

[0020] A preset scene recognition model is applied to perform scene recognition processing on the target image, a scene identifier output by the scene recognition model is obtained, and the scene identifier is used as label data of the scene label; the scene recognition model is a model obtained by training the Hiera model.

[0021] In the above method, optionally, the step of converting the tag information into a semantic description form to obtain the tag description information includes:

[0022] Get the preset heuristic prompt template;

[0023] In the heuristic prompt template, determining a prompt position corresponding to each of the label types;

[0024] According to the prompt position corresponding to each of the label types, the label data of each of the label types is filled into the heuristic prompt template to obtain the heuristic prompt template filled with data;

[0025] Based on the heuristic prompt template filled with data, a heuristic prompt sentence is generated, and the heuristic prompt sentence is used as the label description information.

[0026] Optionally, the method further comprises performing unified characterization processing on the target image, the descriptive text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the descriptive text information, and label description features corresponding to the label description information, including:

[0027] By using a preset CLIP model, the target image, the description text information and the label description information are vector-represented to obtain a feature vector corresponding to the target image, a feature vector corresponding to the description text information, and a feature vector corresponding to the label description information;

[0028] Taking the feature vector corresponding to the target image as the image feature;

[0029] Using the feature vector corresponding to the description text information as the text description feature;

[0030] The feature vector corresponding to the label description information is used as the label description feature.

[0031] In the above method, optionally, performing information enhancement processing on the metadata feature and the image feature to obtain metadata enhancement features corresponding to the metadata feature and image enhancement features corresponding to the image feature includes:

[0032] Performing feature embedding processing on the metadata feature and the image feature respectively to obtain metadata embedding features corresponding to the metadata feature and image embedding features corresponding to the image feature;

[0033] Based on a preset multi-head attention mechanism, adaptive correlation learning is performed on the metadata embedding feature and the image embedding feature to optimize the metadata embedding feature and the image embedding feature to obtain optimized features corresponding to the metadata embedding feature and optimized features corresponding to the image embedding feature;

[0034] Using the optimized feature corresponding to the metadata embedding feature as the metadata enhancement feature;

[0035] The optimized feature corresponding to the image embedding feature is used as the image enhancement feature.

[0036] In the above method, optionally, performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain the fusion feature of the target image includes:

[0037] A preset cross-modal transformer is applied to perform feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a cross-modal fusion feature output by the cross-modal transformer, and the cross-modal fusion feature is used as a fusion feature of the target image.

[0038] In the above method, optionally, performing emotion recognition on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image includes:

[0039] Applying a preset classification module to perform emotion category prediction on the fusion features of the target image to obtain the prediction probability of each candidate emotion category;

[0040] The candidate emotion category with the largest predicted probability among the candidate emotion categories is used as the target candidate emotion category, and the target candidate emotion category is used as the emotion category corresponding to the target image.

[0041] An image sentiment analysis device, comprising:

[0042] A metadata acquisition unit, used to determine a metadata set of the target image when sentiment analysis is required for the target image; the metadata set includes descriptive text information and tag information corresponding to the target image; the tag information includes tag data of multiple tag types;

[0043] A label conversion unit, used for converting the label information into a semantic description form to obtain label description information;

[0044] A unified characterization unit, configured to perform unified characterization processing on the target image, the descriptive text information, and the label description information to obtain image features corresponding to the target image, text description features corresponding to the descriptive text information, and label description features corresponding to the label description information;

[0045] a feature enhancement unit, configured to use the text description feature and the tag description feature as metadata features, perform information enhancement processing on the metadata features and the image features, and obtain metadata enhancement features corresponding to the metadata features and image enhancement features corresponding to the image features;

[0046] A feature fusion unit, used for performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a fusion feature of the target image;

[0047] The emotion recognition unit is used to perform emotion recognition on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image.

[0048] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the image sentiment analysis method as described above.

[0049] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors as the above-mentioned image sentiment analysis method.

[0050] An image sentiment analysis method provided based on the above-mentioned embodiment of the present invention includes: when it is necessary to perform sentiment analysis on a target image, determining a metadata set of the target image; the metadata set includes descriptive text information and label information corresponding to the target image; the label information includes label data of multiple label types; performing semantic description form conversion processing on the label information to obtain label description information; performing unified representation processing on the target image, the descriptive text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the descriptive text information, and label description features corresponding to the label description information; using the text description features and the label description features as metadata features, performing information enhancement processing on the metadata features and the image features to obtain metadata enhancement features corresponding to the metadata features, and image enhancement features corresponding to the image features; performing cross-modal feature fusion processing on the metadata enhancement features and the image enhancement features to obtain fusion features of the target image; performing emotion recognition on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image. By applying the method provided in the embodiment of the present invention, the metadata of the image can be used as a feature, and the feature information can be enhanced based on the metadata features and image features of the image, and the metadata features and image features that have been subjected to feature enhancement processing can be cross-modal feature fusion to identify the emotion category of the image based on the fused features. In the process of image emotion analysis, the image metadata and the visual features of the image can be combined for multi-dimensional analysis. Even if there is a certain difference between the visual information of the image and the actual expression, the contextual inference based on the metadata can make the emotion recognition result tend to be a true expression, which is conducive to improving the accuracy of image emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 A method flow chart of an image sentiment analysis method provided by an embodiment of the present invention;

[0053] Figure 2 Another method flow chart of an image sentiment analysis method provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the architecture of an image sentiment analysis system provided by an embodiment of the present invention;

[0055] Figure 4A schematic diagram of the structure of an image sentiment analysis device provided by an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0059] The embodiment of the present invention provides an image sentiment analysis method, which can be applied to various platforms that need to perform sentiment analysis on images. The execution subject can be a server of the platform. The method flow chart of the method is as follows: Figure 1 As shown, including:

[0060] S101: When it is necessary to perform sentiment analysis on a target image, determine a metadata set of the target image; the metadata set includes descriptive text information and tag information corresponding to the target image; the tag information includes tag data of multiple tag types;

[0061] In the method provided in the embodiment of the present invention, the metadata type required to be applied in the image sentiment analysis process can be pre-set. The metadata of an image refers to data that describes the image, such as a text description of the image, various keyword tags, time / space information of the shooting, and other data. These data are usually automatically generated when the image is generated.

[0062] In the method provided by the embodiment of the present invention, when it is necessary to analyze the emotional expression of a certain image, the image can be used as the target image. According to the predetermined metadata acquisition method, various types of metadata corresponding to the target image are determined, specifically including the descriptive text information corresponding to the target image and the label data of each label type, and other types of metadata may also be included. The label data of each label type may include the label data of the object label, the label data of the scene label, and so on. The various types of metadata corresponding to the target image can be obtained from the metadata information carried in the image file of the target image, or the target image can be parsed by other methods to obtain the corresponding type of metadata. The specific metadata acquisition method can be set according to actual needs, without affecting the function of the method provided by the embodiment of the present invention.

[0063] It should be noted that, in the method provided in the embodiment of the present invention, the metadata obtained from the metadata information of the target image is obtained with authorization.

[0064] S102: performing semantic description conversion processing on the tag information to obtain tag description information;

[0065] In the method provided by the embodiment of the present invention, the label information is label data of each label type, and the label data is usually a phrase. In order to align various metadata, the label information needs to be converted into a semantic description form. Specifically, the various label data in the label information are converted from the description form of a phrase to the description form of a sentence. That is, based on the various label data in the label information, sentences describing the various label data are generated, and the sentences describing the various label data are used as label description information.

[0066] S103: performing unified characterization processing on the target image, the description text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the description text information, and label description features corresponding to the label description information;

[0067] In the method provided by the embodiment of the present invention, according to a predetermined representation method, the target image, the descriptive text information and the label description information are characterized in a unified manner to obtain image features corresponding to the target image, obtain text description features corresponding to the descriptive text information, and obtain label description features corresponding to the label description information. For example, the target image, the descriptive text information and the label description information can be represented by vectors, and the above objects can be mapped to the same vector space, with the elements mapped by each object in the same vector space as the corresponding features.

[0068] S104: taking the text description feature and the tag description feature as metadata features, performing information enhancement processing on the metadata features and the image features, and obtaining metadata enhancement features corresponding to the metadata features and image enhancement features corresponding to the image features;

[0069] In the method provided in the embodiment of the present invention, the text description features and the label description features are collectively referred to as metadata features. According to a predetermined feature enhancement method, the metadata features and the image features are enhanced to extract more effective features from the metadata features and the image features, thereby optimizing the features. The features obtained after the metadata features are enhanced are used as metadata enhancement features, and the features obtained after the image features are enhanced are used as image enhancement features. The feature enhancement in the embodiment of the present invention mainly extracts features that are more strongly correlated with emotion recognition on the basis of existing features to optimize the features. The feature enhancement method can be set according to actual needs, and can be implemented using existing feature enhancement technologies, or other methods can be configured as needed.

[0070] S105: performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a fusion feature of the target image;

[0071] In the method provided in the embodiment of the present invention, a cross-modal feature fusion mechanism for fusing different types of feature data can be pre-set. In the process of image emotion recognition, the metadata enhancement features and the image enhancement features are fused through the preset cross-modal feature fusion mechanism, and the features obtained by fusing the metadata enhancement features and the image enhancement features are used as the fusion features of the target image. The cross-modal feature fusion mechanism can be deployed based on a deep neural network, or it can be set using an attention mechanism, or other methods can be selected as needed to perform cross-modal feature fusion, which does not affect the function of the method provided in the embodiment of the present invention.

[0072] S106: Performing emotion recognition on the target image based on the fusion features of the target image to obtain an emotion category corresponding to the target image.

[0073] In the method provided by the embodiment of the present invention, a classification algorithm can be deployed in advance according to actual needs, and the classification algorithm can be set using the existing classification algorithm principle. In the process of image sentiment analysis, the fusion features of the target image can be used as feature data, and the target image can be sentimentally classified based on the fusion features of the target image through the classification algorithm, so as to obtain the sentiment category corresponding to the target image, and the sentiment category is used as the sentiment analysis result of the target image.

[0074] Based on the method provided by the embodiment of the present invention, when it is necessary to perform sentiment analysis on the target image, a metadata set of the target image is determined; the metadata set includes description text information and label information corresponding to the target image; the label information includes label data of multiple label types; the label information is converted into a semantic description form to obtain label description information; the target image, the description text information and the label description information are uniformly characterized to obtain image features corresponding to the target image, text description features corresponding to the description text information, and label description features corresponding to the label description information; the text description features and the label description features are used as metadata features, and information enhancement processing is performed on the metadata features and the image features to obtain metadata enhancement features corresponding to the metadata features, and image enhancement features corresponding to the image features; the metadata enhancement features and the image enhancement features are cross-modal feature fusion processed to obtain the fusion features of the target image; the target image is emotion recognized based on the fusion features of the target image to obtain the emotion category corresponding to the target image. By applying the method provided by the embodiment of the present invention, the metadata of the image can be used as a feature, feature information can be enhanced on the basis of the metadata features and image features of the image, and cross-modal feature fusion is performed on the metadata features and image features after feature enhancement processing, and the emotion category of the image can be recognized based on the fused features. In the process of image sentiment analysis, image metadata and image visual features can be combined for multi-dimensional analysis. Even if there is a certain difference between the visual information of the image and the actual expression, contextual inference based on metadata can make the sentiment recognition result tend to be true expression, which is conducive to improving the accuracy of image sentiment recognition.

[0075] exist Figure 1 Based on the method shown, Figure 2 As shown, in the method provided by the embodiment of the present invention, each of the tag types mentioned in step S101 includes an object tag and a scene tag, and the process of determining the metadata set of the target image mentioned in step S101 includes:

[0076] S201: Reading metadata information of the target image;

[0077] In the method provided by the embodiment of the present invention, metadata can be first read from the image file of the target image, that is, metadata of a specified metadata type can be read from the file information of the target image to obtain metadata information. In a specific application scenario, the file data of the target image may contain complete and readable metadata, or some metadata may be missing, or metadata may not be disclosed due to permission restrictions. Therefore, in the process of reading metadata information, all required metadata may be read, or no required metadata may be read, or only part of the metadata may be read, that is, the metadata information may contain metadata, or may be null.

[0078] S202: Determine whether the metadata information contains target attribute data; the target attribute data includes original image description text, original object label data and original scene label data;

[0079] In the method provided in the embodiment of the present invention, the metadata types to be obtained include image description text (i.e., text description Caption of the image), object tags and scene tags. Object tags and scene tags are two types of keyword tags in image metadata. Object tags refer to the object information contained in the image, such as various types of objects in the image, and scene tags refer to the scene category of the image, such as indoor, outdoor, urban, rural, etc.

[0080] In the method provided by the embodiment of the present invention, the read metadata information can be parsed to identify whether it contains target attribute data, namely, the original image description text, original object label data and original scene label data of the target image. The original image description text is the image description text carried in the image file of the target image, the original object label data is the label data of the object label carried in the image file of the target image, and the original scene label data is the label data of the scene label carried in the image file of the target image.

[0081] S203: If the metadata information does not include the target attribute data, a preset image description model is applied to perform image description processing on the target image, and an image description text output by the image description model is obtained, and the image description text is used as the description text information corresponding to the target image; the image description model is a model obtained by training a BLIP model;

[0082] In the method provided by the embodiment of the present invention, if the metadata information read does not contain the original image description text, original object label data and original scene label data of the target image, a pre-deployed model is called to generate the required metadata.

[0083] In the method provided in the embodiment of the present invention, a BLIP (Bootstrapping Language-Image Pre-training) model is pre-trained based on the COCO (Common Objects in Context) dataset, and the training task is to generate text descriptions based on images, so that the BLIP model can generate text descriptions based on images. The COCO dataset is an existing image dataset. The BLIP model can be called a bootstrap language-image pre-training model, which is a pre-training model designed to learn unified representations of vision and text. It is an existing pre-training model and is not described in detail here. The trained BLIP model is used as an image description model.

[0084] In the process of image sentiment analysis, if the read metadata information does not contain the original image description text of the image, the target image is input into the preset image description model, and the image description model describes the target image and outputs the image description text corresponding to the target image. The image description text output by the image description model is used as the description text information corresponding to the target image.

[0085] S204: applying a preset object recognition model to perform object recognition processing on the target image, obtaining each object identifier output by the object recognition model, and using each object identifier as label data of the object label; the object recognition model is a model obtained by training a fast regional convolutional neural network;

[0086] In the method provided by the embodiment of the present invention, a fast regional convolutional neural network (Faster R-CNN) can be pre-constructed, and the Faster R-CNN network is trained based on a sample data set. The training task is to identify the target object in the image. The sample data set contains multiple groups of training samples, each group of samples contains a sample image and annotation data about the target object in the image. The existing image data set can be used as the sample data. The trained fast regional convolutional neural network is used as an object recognition model.

[0087] In the process of image sentiment analysis, if the read metadata information does not contain the original object label data of the image, the target image is input into a preset object recognition model, and the object recognition model performs object recognition on the target image. The object recognition model can recognize the objects that may be contained in the target image based on the target image, and sort the objects according to the confidence level, and take the top k objects as the recognition results, and output the object identifiers of these objects, where k can be set according to actual needs. The object identifiers output by the object recognition model are used as the label data of the object label corresponding to the target image.

[0088] S205: Apply a preset scene recognition model to perform scene recognition processing on the target image, obtain a scene identifier output by the scene recognition model, and use the scene identifier as label data of the scene label; the scene recognition model is a model obtained by training the Hiera model.

[0089] In the method provided in the embodiment of the present invention, the Hiera model is pre-trained based on the Place365 dataset, and the training task is to classify the image into scenes, that is, to identify the scene category of the image. The Place365 dataset is an existing scene recognition dataset. The Hiera model can be called a hierarchical visual representation model, which is a simplified hierarchical visual Transformer model. Its model architecture is an existing model architecture and is not described in detail here. The trained Hiera model is used as a scene recognition model.

[0090] In the process of image sentiment analysis, if the read metadata information does not contain the original scene label data of the image, the target image is input into the preset scene recognition model, and the scene recognition model performs scene recognition on the target image, that is, scene classification on the target image. The scene recognition model can identify the probability of the target image in each scene category based on the target image, and use the scene category with the highest probability as the scene identifier corresponding to the target image, and output the scene identifier corresponding to the target image, that is, the scene category to which the target image belongs. The scene identifier output by the scene recognition model is used as the label data of the scene label corresponding to the target image.

[0091] It can be understood that in the method provided by the embodiment of the present invention, the label data of the object label and the label data of the scene label are the label data of each label type, and the label data of the object label and the label data of the scene label are used as the label information corresponding to the target image, and then a metadata set containing descriptive text information and label information is obtained.

[0092] Furthermore, in the method provided by the embodiment of the present invention, if during the judgment process of step S202, it is determined that the metadata information contains the target attribute data, that is, contains the original image description text, the original object label data and the original scene label data, then the original image description text can be used as the descriptive text information corresponding to the target image, the original object label data can be used as the label data of the object label corresponding to the target image, and the original scene label data can be used as the label data of the scene label corresponding to the target image, thereby obtaining a metadata set of the target image.

[0093] It should be noted that in the actual application process, the metadata information read may contain some data in the target attribute data, but lack other data. At this time, the data contained in the metadata information can be used as the corresponding data in the metadata set. For the data type not included, the corresponding model can be called to generate data.

[0094] Based on the method provided in the embodiment of the present invention, metadata can be read from the image file first. In the case of missing metadata, the corresponding model can be applied to generate corresponding metadata. Regardless of whether the image file can provide the required metadata, metadata features can be obtained, which is conducive to analyzing images in various application environments.

[0095] exist Figure 1 On the basis of the method shown, in the method provided by the embodiment of the present invention, the process of converting the label information into a semantic description form to obtain the label description information mentioned in step S102 includes:

[0096] Get the preset heuristic prompt template;

[0097] In the heuristic prompt template, determining a prompt position corresponding to each of the label types;

[0098] In the method provided in the embodiment of the present invention, heuristic hint learning is used to represent label information. Specifically, corresponding heuristic hint templates can be set in advance according to each label type. For example, each label type includes object labels and scene labels. The label data of the object label is marked as T obj , T obj =obj1,obj2,…,obj k , obj represents a specific label, and the label data of the scene label is marked as T sce The heuristic hint template P can be set to “the scene or background of the image is T sce , and the image contains the following objects: obj1,obj2,…,obj k ”, which means “the scene or background of the image is T sce , and the image contains the following objects: obj1,obj2,…,obj k ". It is understandable that the T in the heuristic prompt template sce and obj only represent the data object corresponding to the position, which can also be understood as a placeholder rather than the actual label data.

[0099] During the image analysis process, a preset heuristic prompt template can be obtained, and the prompt position corresponding to each label type can be located according to the content in the template. For example, in the template P shown above, T sce The current location is the prompt location corresponding to the scene label.

[0100] According to the prompt position corresponding to each of the label types, the label data of each of the label types is filled into the heuristic prompt template to obtain the heuristic prompt template filled with data;

[0101] Based on the heuristic prompt template filled with data, a heuristic prompt sentence is generated, and the heuristic prompt sentence is used as the label description information.

[0102] In the method provided by the embodiment of the present invention, the label data of the corresponding label type is filled in the prompt position corresponding to each label type in the heuristic prompt template, and a heuristic prompt sentence is generated based on the heuristic prompt template filled with the label data, and the heuristic prompt sentence is used as the label description information. For example, the template P shown above replaces the T sce By replacing the label data of the scene label in the metadata set of the target image, and replacing each obj object with the label data of the object label in the metadata set of the target image, a heuristic prompt template filled with data can be obtained, and a heuristic prompt statement can be derived based on the template.

[0103] exist Figure 1 On the basis of the method shown in the figure, in the method provided by the embodiment of the present invention, the process of uniformly characterizing the target image, the descriptive text information and the label description information mentioned in step S103 to obtain the image features corresponding to the target image, the text description features corresponding to the descriptive text information, and the label description features corresponding to the label description information includes:

[0104] By using a preset CLIP model, the target image, the description text information and the label description information are vector-represented to obtain a feature vector corresponding to the target image, a feature vector corresponding to the description text information, and a feature vector corresponding to the label description information;

[0105] In the method provided in the embodiment of the present invention, unified feature representation is achieved by mapping various types of feature data to the same vector space, and a CLIP (Contrastive Language-Image Pre-training) model is specifically set up. The CLIP model can be called a contrastive learning language-image pre-training model, which is a type of existing multimodal pre-training model and is not introduced in detail here. The CLIP model can map images and texts to the same vector space through contrastive learning. Specifically, in the method provided in the embodiment of the present invention, the image encoder based on the ViT-B / 32 architecture and the text encoder based on the masked self-attention Transformer architecture in the CLIP model are used to align and uniformly represent images and metadata.

[0106] During the image analysis process, the target image, the descriptive text information corresponding to the target image, and the label description information can be input into a preset CLIP model. The CLIP model maps the target image, the descriptive text information, and the label description information to the same vector space through comparative learning, and obtains the element mapped by the target image in the vector space, that is, the feature vector corresponding to the target image, obtains the element mapped by the descriptive text information in the vector space, that is, the feature vector corresponding to the descriptive text information, and obtains the element mapped by the label description information in the vector space, that is, the feature vector corresponding to the label description information.

[0107] Taking the feature vector corresponding to the target image as the image feature;

[0108] Using the feature vector corresponding to the description text information as the text description feature;

[0109] The feature vector corresponding to the label description information is used as the label description feature.

[0110] In the method provided by the embodiment of the present invention, the feature vector corresponding to the target image is used as the image feature corresponding to the target image. The feature vector corresponding to the description text information is used as the text description feature corresponding to the description text information. The feature vector corresponding to the label description information is used as the label description feature corresponding to the label description information.

[0111] Based on the method provided in the embodiment of the present invention, the CLIP model can be applied to perform unified vector representation of images and metadata. The CLIP model has good unified representation capabilities for images and texts, which is beneficial to improving the accuracy of image and metadata feature expression and further improving the accuracy of sentiment analysis.

[0112] exist Figure 1 On the basis of the method shown, in the method provided by the embodiment of the present invention, the process of performing information enhancement processing on the metadata feature and the image feature mentioned in step S104 to obtain the metadata enhancement feature corresponding to the metadata feature and the image enhancement feature corresponding to the image feature includes:

[0113] Performing feature embedding processing on the metadata feature and the image feature respectively to obtain metadata embedding features corresponding to the metadata feature and image embedding features corresponding to the image feature;

[0114] In the method provided by the embodiment of the present invention, in the process of feature information enhancement, firstly, feature embedding is performed on metadata features (text description features and label description features) and image features respectively. Specifically, a parallel fully connected (FC) layer and a Transformer network structure can be used to perform feature embedding processing on metadata features and image features respectively to obtain metadata features processed by feature embedding (i.e., metadata embedded features) and image features processed by feature embedding (i.e., image embedded features).

[0115] Based on a preset multi-head attention mechanism, adaptive correlation learning is performed on the metadata embedding feature and the image embedding feature to optimize the metadata embedding feature and the image embedding feature to obtain optimized features corresponding to the metadata embedding feature and optimized features corresponding to the image embedding feature;

[0116] In the method provided by the embodiment of the present invention, a multi-head attention mechanism is used to perform adaptive correlation learning on metadata features and image features, so as to highlight more effective information in metadata features and image features while suppressing parts with weaker information.

[0117] In the method provided by the embodiment of the present invention, multiple cascaded Transformer layers can be constructed as an optimization network for image features. By training the optimization network for image features, it can capture the features of the image from low level to high level, so as to optimize the visual features of the image. In the image analysis process, the optimization network for image features can be applied to optimize the image embedding features step by step. At the same time, various types of metadata in the metadata embedding features are processed through a multi-head attention mechanism, and the image features obtained by step-by-step optimization are interactively learned with the metadata embedding features. The image features are used as queries, and the metadata embedding features are used as keys and values. Thus, the metadata embedding features are continuously optimized through the correlation between the metadata and the image features, and finally the optimized image embedding features (i.e., the optimized features corresponding to the image embedding features) and the optimized metadata embedding features (i.e., the optimized features corresponding to the metadata embedding features) are obtained.

[0118] Using the optimized feature corresponding to the metadata embedding feature as the metadata enhancement feature;

[0119] The optimized feature corresponding to the image embedding feature is used as the image enhancement feature.

[0120] In the method provided by the embodiment of the present invention, the optimized metadata embedding feature is used as the metadata enhancement feature, and the optimized image embedding feature is used as the image enhancement feature.

[0121] Based on the method provided in the embodiment of the present invention, metadata features and image features can be optimized through adaptive correlation learning, which is conducive to obtaining feature data with strong correlation with image analysis, thereby improving the accuracy of image sentiment analysis.

[0122] exist Figure 1 On the basis of the method shown, in the method provided by the embodiment of the present invention, the process of performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature mentioned in step S105 to obtain the fusion feature of the target image includes:

[0123] A preset cross-modal transformer is applied to perform feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a cross-modal fusion feature output by the cross-modal transformer, and the cross-modal fusion feature is used as a fusion feature of the target image.

[0124] In the method provided by the embodiment of the present invention, a cross-modal transformer can be pre-deployed, and the cross-modal transformer can be formed by deeply stacking multiple cross-modal attention blocks. By training the cross-modal transformer, it can capture and fuse information from different types of data. In the process of image sentiment analysis, the metadata enhancement features and image enhancement features can be input into the preset cross-modal transformer, and the input features can be fused by the cross-modal transformer. The cross-modal fusion features obtained by the cross-modal transformer fusing the metadata enhancement features and the image enhancement features are used as the fusion features of the target image.

[0125] exist Figure 1 On the basis of the method shown, in the method provided by the embodiment of the present invention, the process of performing emotion recognition on the target image based on the fusion features of the target image mentioned in step S106 to obtain the emotion category corresponding to the target image includes:

[0126] Applying a preset classification module to perform emotion category prediction on the fusion features of the target image to obtain the prediction probability of each candidate emotion category;

[0127] In the method provided by the embodiment of the present invention, a classification head can be pre-constructed as a classification module, and the classification head can be composed of a linear layer and a softmax (normalized exponential function) layer. In the image sentiment analysis process, the fusion features of the target image can be input into the classification module, and the classification module predicts the probability that the target image belongs to each candidate sentiment category based on the fusion features, and outputs the predicted probability of each candidate sentiment category. The predicted probability of each candidate sentiment category is the probability that the target image belongs to the candidate sentiment category.

[0128] The candidate emotion category with the largest predicted probability among the candidate emotion categories is used as the target candidate emotion category, and the target candidate emotion category is used as the emotion category corresponding to the target image.

[0129] In the method provided by the embodiment of the present invention, the predicted probabilities of each candidate emotion category can be compared to determine the candidate emotion category with the highest predicted probability, and the candidate emotion category with the highest predicted probability can be used as the target candidate emotion category, that is, the emotion category to which the target image belongs.

[0130] In order to better illustrate the method provided by the embodiment of the present invention, based on the methods provided by the above embodiments and in combination with actual application scenarios, the embodiment of the present invention provides another image sentiment analysis method, which can be implemented by constructing a corresponding image sentiment analysis system (also understood as an image sentiment analysis model). The schematic diagram of the architecture of the image sentiment analysis system provided by the embodiment of the present invention can be shown as follows: Figure 3 As shown, the system mainly includes a feature representation module 301, an adaptive correlation learning module 302 and a cross-modal fusion and prediction module 303.

[0131] When sentiment analysis of a target image is required, the system needs to obtain metadata related to the target image, including image description text, scene label data, and object label data. First, the system can read metadata from the image file of the target image. For metadata that can be read, the data carried in the image file is directly applied. If there is missing metadata, the pre-trained model is applied to generate the corresponding metadata. For example, the BLIP model can be used to generate description text for the target image, the fast regional convolutional neural network can be used to identify objects in the target image to obtain object label data, and the Hiera model can be applied to identify the scene of the target image to obtain scene label data.

[0132] The system can input the target image, the descriptive text of the target image, and various labels of the target image into the feature representation module, and the feature representation module represents the image and metadata. In order to achieve input alignment and better represent multiple metadata, the label data such as the object label and scene label of the target image are first converted into a heuristic prompt sentence through the heuristic prompt learning technology. The specific heuristic prompt sentence can refer to the example P of the heuristic prompt template mentioned in the previous embodiment. Then, the CLIP model is used to uniformly represent the target image and various metadata. In the embodiment of the present invention, the letter I represents the target image, the letter C represents the descriptive text of the target image, and the letter P represents the heuristic prompt sentence describing various types of label data. Using the image encoder based on the ViT-B / 32 architecture and the text encoder based on the masked self-attention Transformer architecture in the CLIP model, the image and metadata are mapped to a shared semantic space to uniformly represent I, C and P, and generate a representation E of I. v , characterization of C c , characterization of P p , each representation generated is a 512-dimensional vector, which can be formally expressed as the following formula 1:

[0133] E v ,E c ,E p =CLIP(I,C,P) (Formula 1)

[0134] in, , , feature dimension d e =512.

[0135] The feature vector of the target image (E v ) and the feature vector of metadata (the feature vector E describing the text c and the feature vector E of the heuristic prompt sentence p ) is input into the adaptive correlation learning module to enhance the information of image features and metadata features.

[0136] The adaptive correlation learning module first embeds the feature vectors of various features and uses image features to supervise and correct metadata features. Specifically, the fully connected layer (FC layer) and the parallel Transformer layer are first used to embed the feature data. The specific principle can be formally expressed as the following equations 2 and 3:

[0137] (Formula 2)

[0138] (Formula 3)

[0139] in, , represents the output data of the FC layer. , represents the output data of the Transformer layer. , , these two parameters are trainable parameters. , d h =128, L=8.

[0140] In order to adaptively learn the appropriate weight for each metadata, a multi-head attention mechanism is used in the adaptive relevance learning module to highlight more effective feature information while suppressing parts with weaker information. Specifically, in order to capture the features of the image from low-level to high-level, first, the features of the target image are learned and optimized through multiple cascaded Transformer layers. The way to optimize the image features can be formally expressed as the following formula 4:

[0141] (Formula 4)

[0142] Among them, j=1,…,N, N is the number of Transformer layers used to optimize image features.

[0143] Then, through the multi-head attention mechanism, the continuously optimized image features interact with the metadata features, and the metadata features are optimized through adaptive correlation learning. In the interaction between images and metadata, the image feature sequence is used as the query, and the embedded features of the metadata are used as the key and value. In order to learn the joint representation of multiple metadata, a new tag is randomly initialized. At the same time, the residual block is introduced to gradually accumulate and update the learned metadata representation The adaptive correlation learning process of metadata features can be formalized by the following formulas:

[0144] (Formula 5)

[0145] (Formula 6)

[0146] (Formula 7)

[0147] (Formula 8)

[0148] Where j = 1,…,N,d k represents the dimension of each head in the multi-head attention, MHA represents the multi-head attention model, Q, K, V represent the Q (query) matrix, K (key) matrix and V (value) matrix in the multi-head attention mechanism, respectively, W Q , W K , W V and W OThese are all trainable parameters.

[0149] After being processed by the adaptive correlation learning module, the image features with feature enhancement can be obtained. and feature-enhanced metadata features .

[0150] After obtaining the enhanced features of the image and metadata, the system fuses the optimized metadata features and image features through the cross-modal fusion and prediction module, and performs sentiment prediction based on the fused features. In the embodiment of the present invention, a cross-modal transformer is used to combine image features and metadata features to capture interactive information from different modalities. First, the image features and metadata features after feature enhancement are taken as input, and an additional tag H is added. e and position embedding, H e is a h The dimensional tag is used to capture the global information of the sequence head, while the position embedding is used to preserve the sequence information of the input. The cross-modal transformer is a deep stack of multiple cross-modal attention blocks with a depth of M. In the cross-modal transformer, As a query, As keys and values. The processing principle of the cross-modal transformer for features can be formally expressed as the following formulas:

[0151] (Formula 9)

[0152] (Formula 10)

[0153] In each conversion block, the multi-head attention calculates the attention from the target to the source as follows:

[0154] (Formula 11)

[0155] (Formula 12)

[0156] in, represents the join operation along the sequence dimension, Represents the added token tag. Position embedding is defined as as well as . , , , are all trainable parameters. Finally, after repeating the processing of M times of deep stacked cross-modal Transformer layers, the final output can be obtained, which is the fusion feature of image features and metadata features. .

[0157] Finally, the system performs sentiment classification on the fused features. Specifically, a classification head (linear layer + Softmax layer) is deployed in the cross-modal fusion and prediction module, and the classification head is used for final prediction. In the sentiment classification process, the first tag of the sequence is first selected from the fused features output by the cross-modal transformer. , the tag at the head of the sequence usually contains the global information of the entire sequence. Then the classification header is applied to process the tag. The processing of the classification header can be formally expressed as follows:

[0158] (Formula 13)

[0159] Among them, p is the output of the classification head, and the output of the classification head is the probability distribution of all candidate emotion categories. , , these two parameters are trainable parameters.

[0160] According to the probability distribution of each candidate emotion category output by the classification head, the candidate emotion category with the highest probability can be selected as the final emotion analysis result, that is, the emotion category of the target image.

[0161] In the embodiment of the present invention, during the training process of the system, a cross entropy loss function can be used as a loss function to optimize the accuracy of sentiment classification. The calculation principle of the loss L can be shown as follows:

[0162] (Formula 14)

[0163] Among them, y i represents the true sentiment label of the i-th sample data in the sample data set, P(p i | I) represents the image sentiment analysis result of the system for the i-th sample data, that is, the probability distribution of the sentiment category predicted for the sample image, and n represents the number of sample data in the sample data set.

[0164] In general, in the method provided by the embodiment of the present invention, a variety of metadata corresponding to the image are first obtained, and each type of metadata is aligned using heuristic prompt learning, and the original representation of the image and text is uniformly generated using a pre-trained model. Then, the original representation of the image and its related metadata is combined, and a better low-dimensional implicit feature is obtained through a fully connected layer. Adaptive correlation learning is used to highlight the more effective representations of various features, while suppressing the representations with weaker information, and the image representation is used as a supervision vector to enhance the metadata representation. Multimodal feature fusion is performed on the enhanced image features and metadata features, and the fused features are classified using a classification head, and the probability distribution of each candidate emotion category is output, and the category with the highest probability is selected as the prediction result of the emotion classification.

[0165] Based on the method provided by the embodiment of the present invention, the visual information of the image and the related metadata information can be fully utilized for emotion recognition, thereby enhancing the emotional understanding of the image. Through the fusion of multimodal features and cross-modal learning, the emotions conveyed by the image can be more accurately predicted, which is conducive to improving the accuracy of image emotion analysis.

[0166] and Figure 1 Corresponding to the image emotion analysis method shown in FIG. 1 , an embodiment of the present invention further provides an image emotion analysis device for analyzing Figure 1 The specific implementation of the method shown in is shown in the structural diagram Figure 4 As shown, including:

[0167] The metadata acquisition unit 401 is used to determine the metadata set of the target image when it is necessary to perform sentiment analysis on the target image; the metadata set includes description text information and tag information corresponding to the target image; the tag information includes tag data of multiple tag types;

[0168] The label conversion unit 402 is used to convert the label information into a semantic description form to obtain label description information;

[0169] A unified characterization unit 403 is used to perform unified characterization processing on the target image, the description text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the description text information, and label description features corresponding to the label description information;

[0170] A feature enhancement unit 404 is used to use the text description feature and the tag description feature as metadata features, perform information enhancement processing on the metadata features and the image features, and obtain metadata enhancement features corresponding to the metadata features and image enhancement features corresponding to the image features;

[0171] A feature fusion unit 405 is used to perform cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a fusion feature of the target image;

[0172] The emotion recognition unit 406 is used to perform emotion recognition on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image.

[0173] By using the device provided by the embodiment of the present invention, the metadata of the image can be used as a feature, and feature information can be enhanced based on the metadata features and image features of the image, and cross-modal feature fusion can be performed on the metadata features and image features that have undergone feature enhancement processing, and the emotion category of the image can be identified based on the fused features. In the process of image emotion analysis, multi-dimensional analysis can be performed in combination with the image metadata and the visual features of the image. Even if there is a certain difference between the visual information of the image and the actual expression, the contextual inference based on the metadata can make the emotion recognition result tend to be a true expression, which is conducive to improving the accuracy of image emotion recognition.

[0174] exist Figure 4 Based on the device shown, the device provided by the embodiment of the present invention can be further expanded into multiple units. The function of each unit can be found in the description of each embodiment provided for the image sentiment analysis method above, and no further examples will be given here.

[0175] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the image sentiment analysis method as described above.

[0176] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to perform the following operations:

[0177] When it is necessary to perform sentiment analysis on a target image, a metadata set of the target image is determined; the metadata set includes descriptive text information and tag information corresponding to the target image; the tag information includes tag data of multiple tag types;

[0178] Performing semantic description conversion processing on the tag information to obtain tag description information;

[0179] Performing unified characterization processing on the target image, the description text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the description text information, and label description features corresponding to the label description information;

[0180] The text description feature and the tag description feature are used as metadata features, and information enhancement processing is performed on the metadata features and the image features to obtain metadata enhancement features corresponding to the metadata features and image enhancement features corresponding to the image features;

[0181] Performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a fusion feature of the target image;

[0182] Emotion recognition is performed on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image.

[0183] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0184] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0185] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image sentiment analysis, characterized in that: include: When it is necessary to perform sentiment analysis on a target image, determining a metadata set of the target image; The metadata set includes description text information and tag information corresponding to the target image; the tag information includes tag data of multiple tag types; Performing semantic description conversion processing on the tag information to obtain tag description information; Performing unified characterization processing on the target image, the description text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the description text information, and label description features corresponding to the label description information; The text description feature and the tag description feature are used as metadata features, and information enhancement processing is performed on the metadata features and the image features to obtain metadata enhancement features corresponding to the metadata features and image enhancement features corresponding to the image features; Performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a fusion feature of the target image; Performing emotion recognition on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image; Wherein, each of the tag types includes an object tag and a scene tag, and the step of determining the metadata set of the target image includes: Reading metadata information of the target image; Determining whether the metadata information contains target attribute data; the target attribute data includes original image description text, original object label data and original scene label data; If the metadata information does not contain the target attribute data, a preset image description model is applied to perform image description processing on the target image, and an image description text output by the image description model is obtained, and the image description text is used as the description text information corresponding to the target image; the image description model is a model obtained by training a BLIP model; Applying a preset object recognition model to perform object recognition processing on the target image, obtaining each object identifier output by the object recognition model, and using each object identifier as label data of the object label; the object recognition model is a model obtained by training a fast regional convolutional neural network; A preset scene recognition model is applied to perform scene recognition processing on the target image, a scene identifier output by the scene recognition model is obtained, and the scene identifier is used as label data of the scene label; the scene recognition model is a model obtained by training the Hiera model.

2. The image sentiment analysis method according to claim 1, characterized in that: The step of converting the label information into a semantic description form to obtain label description information includes: Get the preset heuristic prompt template; In the heuristic prompt template, determining a prompt position corresponding to each of the label types; According to the prompt position corresponding to each of the label types, the label data of each of the label types is filled into the heuristic prompt template to obtain the heuristic prompt template filled with data; Based on the heuristic prompt template filled with data, a heuristic prompt sentence is generated, and the heuristic prompt sentence is used as the label description information.

3. The image sentiment analysis method according to claim 1, characterized in that: The unified characterization processing of the target image, the description text information and the label description information to obtain image features corresponding to the target image, text description features corresponding to the description text information, and label description features corresponding to the label description information includes: By using a preset CLIP model, the target image, the description text information and the label description information are vector-represented to obtain a feature vector corresponding to the target image, a feature vector corresponding to the description text information, and a feature vector corresponding to the label description information; Taking the feature vector corresponding to the target image as the image feature; Using the feature vector corresponding to the description text information as the text description feature; The feature vector corresponding to the label description information is used as the label description feature.

4. The image sentiment analysis method according to claim 1, characterized in that: The performing information enhancement processing on the metadata feature and the image feature to obtain a metadata enhancement feature corresponding to the metadata feature and an image enhancement feature corresponding to the image feature includes: Performing feature embedding processing on the metadata feature and the image feature respectively to obtain metadata embedding features corresponding to the metadata feature and image embedding features corresponding to the image feature; Based on a preset multi-head attention mechanism, adaptive correlation learning is performed on the metadata embedding feature and the image embedding feature to optimize the metadata embedding feature and the image embedding feature to obtain optimized features corresponding to the metadata embedding feature and optimized features corresponding to the image embedding feature; Using the optimized feature corresponding to the metadata embedding feature as the metadata enhancement feature; The optimized feature corresponding to the image embedding feature is used as the image enhancement feature.

5. The image sentiment analysis method according to claim 1, characterized in that: The performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain the fusion feature of the target image includes: A preset cross-modal transformer is applied to perform feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a cross-modal fusion feature output by the cross-modal transformer, and the cross-modal fusion feature is used as a fusion feature of the target image.

6. The image sentiment analysis method according to claim 1, characterized in that: The performing emotion recognition on the target image based on the fusion features of the target image to obtain the emotion category corresponding to the target image includes: Applying a preset classification module to perform emotion category prediction on the fusion features of the target image to obtain the prediction probability of each candidate emotion category; The candidate emotion category with the largest predicted probability among the candidate emotion categories is used as the target candidate emotion category, and the target candidate emotion category is used as the emotion category corresponding to the target image.

7. An image sentiment analysis device, characterized in that: include: A metadata acquisition unit, used to determine a metadata set of the target image when sentiment analysis is required for the target image; The metadata set includes description text information and tag information corresponding to the target image; the tag information includes tag data of multiple tag types; A label conversion unit, used for converting the label information into a semantic description form to obtain label description information; A unified characterization unit, configured to perform unified characterization processing on the target image, the descriptive text information, and the label description information to obtain image features corresponding to the target image, text description features corresponding to the descriptive text information, and label description features corresponding to the label description information; a feature enhancement unit, configured to use the text description feature and the tag description feature as metadata features, perform information enhancement processing on the metadata features and the image features, and obtain metadata enhancement features corresponding to the metadata features and image enhancement features corresponding to the image features; A feature fusion unit, used for performing cross-modal feature fusion processing on the metadata enhancement feature and the image enhancement feature to obtain a fusion feature of the target image; An emotion recognition unit, used to perform emotion recognition on the target image based on the fusion features of the target image to obtain an emotion category corresponding to the target image; Wherein, each of the tag types includes an object tag and a scene tag, and the metadata acquisition unit is specifically used to: Reading metadata information of the target image; Determining whether the metadata information contains target attribute data; the target attribute data includes original image description text, original object label data and original scene label data; If the metadata information does not contain the target attribute data, a preset image description model is applied to perform image description processing on the target image, and an image description text output by the image description model is obtained, and the image description text is used as the description text information corresponding to the target image; the image description model is a model obtained by training a BLIP model; Applying a preset object recognition model to perform object recognition processing on the target image, obtaining each object identifier output by the object recognition model, and using each object identifier as label data of the object label; the object recognition model is a model obtained by training a fast regional convolutional neural network; A preset scene recognition model is applied to perform scene recognition processing on the target image, a scene identifier output by the scene recognition model is obtained, and the scene identifier is used as label data of the scene label; the scene recognition model is a model obtained by training the Hiera model.

8. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the image sentiment analysis method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: It comprises a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform the image sentiment analysis method as described in any one of claims 1 to 6.

Images