Sentiment analysis method, computer device and storage medium
By constructing a multimodal semantic graph and fusing feature representations using graph convolutional neural networks, the problem of insufficient accuracy in existing multimodal sentiment analysis is solved, achieving more efficient fine-grained sentiment analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TENCENT MUSIC ENTERTAINMENT TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-07-20
- Publication Date
- 2026-06-05
AI Technical Summary
Existing multimodal fine-grained sentiment analysis methods struggle to fully utilize the important features of each modality, resulting in insufficient accuracy in sentiment analysis.
By constructing a multimodal semantic graph, a graph convolutional neural network is used to fuse feature representations of text content, object-related information, and aspect terms to obtain target feature representations for sentiment analysis.
It improves the fusion effect of multimodal information and enhances the accuracy of fine-grained sentiment analysis.
Smart Images
Figure CN117056854B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an emotion analysis method, computer device, and storage medium. Background Technology
[0002] With the development of artificial intelligence technology, multimodal fine-grained sentiment analysis technology has emerged. Fine-grained sentiment analysis differs from conventional sentence-level sentiment analysis; it can determine the sentiment of a given aspect within a sentence. Multimodal refers to the integration or fusion of information from multiple data types, such as text, images, and audio.
[0003] Current sentiment analysis methods directly use attention-based interaction of text and image modal features and pooling results to perform fine-grained sentiment analysis. However, this sequential interaction approach makes it difficult to fully exchange information between multiple modalities, and its representation space for multiple modalities is different, making it difficult to fully utilize the important features of each modality. Therefore, its accuracy in fine-grained sentiment analysis still needs to be improved. Summary of the Invention
[0004] Therefore, it is necessary to provide a sentiment analysis method, computer device, and storage medium to address the aforementioned technical problems.
[0005] Firstly, this application provides a sentiment analysis method. The method includes:
[0006] Acquire the multimodal data to be analyzed; the multimodal data includes text content, non-text content, and aspects of the text content;
[0007] Based on the non-text content, object-related information, including object text tags and their associated knowledge, is obtained; the object text tags are the text tags corresponding to the objects contained in the non-text content.
[0008] Using the aspect items as connection nodes, the text content and the object-related information are connected to construct a multimodal semantic graph containing nodes corresponding to the aspect items, the text content, and the object-related information respectively;
[0009] Obtain the initial feature representations corresponding to each of the aspect items, the text content, and the object-related information;
[0010] The multimodal semantic graph and the initial feature representation are input into a trained graph convolutional neural network to obtain the target feature representation corresponding to the aspect terms output by the graph convolutional neural network.
[0011] Based on the target feature representation, the sentiment analysis results of the aspect items are obtained.
[0012] In one embodiment, obtaining object-related information, including object text tags and their associated knowledge, based on the non-text content includes: inputting the non-text content into a trained object recognition model to obtain object text tags output by the object recognition model; obtaining associated knowledge of the object text tags based on an established knowledge base; and obtaining the object-related information based on the object text tags and their associated knowledge.
[0013] In one embodiment, obtaining the associated knowledge of the object text tag based on the established knowledge base includes: determining the object-related text tags and their relationships in the established knowledge base based on the object text tag; the object-related text tags are text tags in the established knowledge base that are associated with the object text tag; and obtaining the associated knowledge of the object text tag based on the object-related text tags and their relationships.
[0014] In one embodiment, obtaining the initial feature representations corresponding to the aspect, the text content, and the object-related information includes: for the text content, obtaining the context representation of the text content through a pre-trained language model to obtain the initial feature representation of the text content; for the aspect and the object-related information, obtaining the embedding feature representations corresponding to the aspect and the object-related information through the embedding mapping function of the pre-trained language model, and transforming the embedding features corresponding to the aspect and the object-related information to the feature space where the context representation is located through a linear layer mapping function to obtain the initial feature representations corresponding to the aspect and the object-related information.
[0015] In one embodiment, obtaining the sentiment analysis result of the aspect item based on the target feature representation includes: inputting the target feature representation output by the graph convolutional neural network into a trained sentiment classifier; and obtaining the sentiment analysis result of the aspect item based on the sentiment classification result of the aspect item output by the sentiment classifier.
[0016] In one embodiment, the step of connecting the text content and the object-related information using the aspect items as connection nodes to construct a multimodal semantic graph containing nodes corresponding to the aspect items, the text content, and the object-related information includes: determining a first subgraph corresponding to the text content; obtaining a second subgraph containing nodes corresponding to the object text tags and their associated knowledge based on the association relationships in the association knowledge; and connecting the first subgraph and the second subgraph using the aspect items as connection nodes to obtain the multimodal semantic graph.
[0017] In one embodiment, determining the first subgraph corresponding to the text content includes: obtaining the dependency parsing tree of the text content to obtain the first subgraph;
[0018] The step of obtaining a second subgraph containing nodes corresponding to the object text label and its associated knowledge based on the association relationships in the associated knowledge includes: making the connection between the node corresponding to the object text label and the node corresponding to its associated knowledge correspond to the association relationships in the associated knowledge, thereby obtaining the second subgraph;
[0019] The step of using the aspect item as the connection node between the first subgraph and the second subgraph to connect the first subgraph and the second subgraph to obtain the multimodal semantic graph includes: replacing the root node of the first subgraph with the node corresponding to the aspect item, and connecting the node corresponding to the aspect item with the node corresponding to the object text label in the second subgraph to obtain the multimodal semantic graph.
[0020] In one embodiment, the method further includes: acquiring training samples and corresponding labeled sentiment tags; the training samples include multimodal semantic graph samples and corresponding initial feature representation samples; inputting the multimodal semantic graph samples and corresponding initial feature representation samples into a graph convolutional neural network to be trained, and having the graph convolutional neural network to be trained output predicted target feature representations corresponding to aspect terms; training the graph convolutional neural network to be trained based on the predicted target feature representations and real sentiment tags to obtain a trained graph convolutional neural network.
[0021] Secondly, this application provides a sentiment analysis method. The method includes:
[0022] A sentiment analysis interface is provided; the sentiment analysis interface includes a multimodal data input area and a sentiment analysis result display area;
[0023] Receive multimodal data input by the user in the multimodal data input area; the multimodal data includes text content and non-text content;
[0024] When the multimodal data input by the user also includes aspect items, the sentiment analysis results of the aspect items are displayed in the sentiment analysis result display area.
[0025] When the multimodal data input by the user does not contain aspect items, the sentiment analysis results of each aspect item contained in the text content are displayed in the sentiment analysis result display area;
[0026] The sentiment analysis results are obtained according to the method described in any of the preceding methods.
[0027] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0028] Acquire multimodal data to be analyzed; the multimodal data includes text content, non-text content, and aspects in the text content; based on the non-text content, obtain object-related information including object text tags and their associated knowledge; the object text tags are the text tags corresponding to the objects contained in the non-text content; connect the text content and the object-related information with the aspects as connection nodes to construct a multimodal semantic graph containing nodes corresponding to the aspects, the text content, and the object-related information respectively; obtain initial feature representations corresponding to the aspects, the text content, and the object-related information respectively; input the multimodal semantic graph and the initial feature representations into a trained graph convolutional neural network to obtain the target feature representations corresponding to the aspects output by the graph convolutional neural network; obtain the sentiment analysis results of the aspects based on the target feature representations.
[0029] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0030] A sentiment analysis interface is provided; the sentiment analysis interface includes a multimodal data input area and a sentiment analysis result display area; it receives multimodal data input by a user in the multimodal data input area; the multimodal data includes text content and non-text content; when the multimodal data input by the user also includes aspect items, the sentiment analysis result of the aspect items is displayed in the sentiment analysis result display area; when the multimodal data input by the user does not include aspect items, the sentiment analysis result of each aspect item contained in the text content is displayed in the sentiment analysis result display area; wherein, the sentiment analysis result is obtained according to the method described in any of the preceding methods.
[0031] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0032] Acquire multimodal data to be analyzed; the multimodal data includes text content, non-text content, and aspects in the text content; based on the non-text content, obtain object-related information including object text tags and their associated knowledge; the object text tags are the text tags corresponding to the objects contained in the non-text content; connect the text content and the object-related information with the aspects as connection nodes to construct a multimodal semantic graph containing nodes corresponding to the aspects, the text content, and the object-related information respectively; obtain initial feature representations corresponding to the aspects, the text content, and the object-related information respectively; input the multimodal semantic graph and the initial feature representations into a trained graph convolutional neural network to obtain the target feature representations corresponding to the aspects output by the graph convolutional neural network; obtain the sentiment analysis results of the aspects based on the target feature representations.
[0033] Sixthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0034] A sentiment analysis interface is provided; the sentiment analysis interface includes a multimodal data input area and a sentiment analysis result display area; it receives multimodal data input by a user in the multimodal data input area; the multimodal data includes text content and non-text content; when the multimodal data input by the user also includes aspect items, the sentiment analysis result of the aspect items is displayed in the sentiment analysis result display area; when the multimodal data input by the user does not include aspect items, the sentiment analysis result of each aspect item contained in the text content is displayed in the sentiment analysis result display area; wherein, the sentiment analysis result is obtained according to the method described in any of the preceding methods.
[0035] The aforementioned sentiment analysis method, computer equipment, and storage medium acquire multimodal data containing text content, non-text content, and aspects within the text content. Based on the non-text content, object-related information, including object text tags and their associated knowledge, is obtained. Aspects are used as connection nodes to link text content and object-related information, thus constructing a multimodal semantic graph. Initial feature representations corresponding to each aspect, text content, and object-related information are obtained. The multimodal semantic graph and initial feature representations are input into a trained graph convolutional neural network (GCNN). The target feature representations corresponding to the aspect items output by the GCNN are obtained, and the sentiment analysis results for the aspect items are obtained based on these target feature representations. This scheme can find associated knowledge of object text tags, enrich the modal information of non-text content, use aspect items to connect text content and object text tags and their associated knowledge to construct a multimodal semantic graph, and use a graph convolutional neural network to fuse feature representations. This allows all modalities to be in the same space, improving the multimodal fusion effect. Furthermore, by fusing multimodal features on a graph structure containing associated knowledge, the accuracy of fine-grained sentiment analysis is improved. Attached Figure Description
[0036] Figure 1 This is a diagram illustrating the application environment of the sentiment analysis method in the embodiments of this application;
[0037] Figure 2 This is a flowchart illustrating the sentiment analysis method in the embodiments of this application;
[0038] Figure 3 This is a schematic diagram illustrating the object identification process in an embodiment of this application;
[0039] Figure 4 This is a schematic diagram of the knowledge base in an embodiment of this application;
[0040] Figure 5 This is a flowchart illustrating the steps involved in constructing a multimodal semantic graph in an embodiment of this application.
[0041] Figure 6 This is a schematic diagram of the structure of the multimodal semantic graph in an exemplary embodiment of this application;
[0042] Figure 7 This is a schematic diagram of the sentiment analysis method in the embodiments of this application;
[0043] Figure 8 This is a flowchart illustrating the steps of training a graph convolutional neural network in an embodiment of this application.
[0044] Figure 9 This is a flowchart illustrating the sentiment analysis method in another embodiment;
[0045] Figure 10 This is a schematic diagram of the sentiment analysis interface in an embodiment of this application;
[0046] Figure 11(a) is an internal structure diagram of the computer device in an embodiment of this application;
[0047] Figure 11(b) is an internal structural diagram of a computer device in another embodiment of this application. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0049] Explanation of abbreviations and terms used in this application:
[0050] External knowledge: Given a phrase, related words, near-synonyms, and semantically related words appear in the knowledge base, such as knowledge graphs like DBpedia and Freebase.
[0051] Object detection and recognition: finding objects in an image, which can include two processes: detection and recognition. The output of object recognition can be the text label of the object in the image.
[0052] The Transformer encoder is a sequence modeling model composed of a self-attention mechanism and a feedforward neural network. Given a text sequence, the Transformer encoder first uses an embedding mapping function to obtain the embedding representation of the sequence, where each embedding vector is formed by adding the corresponding character vector and position vector. Then, the Transformer encoder uses a context mapping function to convert the embedding representation into a semantic representation vector. Typically, the context mapping function is implemented using a multi-layered stacked neural network.
[0053] BERT: short for Bidirectional Encoder Representation from Transformers, is a pre-trained language model that uses Transformer Encoder blocks for connection. It is a bidirectional encoding model and also uses a masked language model (MLM) to generate deep bidirectional language representations.
[0054] GCN stands for Graph Neural Networks. Unlike conventional convolutional neural networks, GCNs allow features and messages to flow and propagate within a graph network.
[0055] The sentiment analysis method provided in this application can be applied to, for example... Figure 1 In the illustrated application environment, the environment may include a terminal 110 and a server 120. The terminal 110 can communicate with the server 120 via the Internet. A data storage system can store the data that the server 120 needs to process. The data storage system can be integrated onto the server 120 or located in the cloud or on other network servers. The terminal 110 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. The server 120 can be implemented using a standalone server or a server cluster consisting of multiple servers.
[0056] This application provides a sentiment analysis method in terms of sentiment analysis processing and sentiment analysis application. In a first aspect, the sentiment analysis method provided in this application can be executed by a server 120 or a terminal 110. Taking the execution by the server 120 as an example, the server 120 can obtain the multimodal data to be analyzed, and then obtain the sentiment analysis results of the aspects in the text content contained in the multimodal data based on the multimodal data. The aspects can be some or all of the aspects in the text content. In a second aspect, the sentiment analysis method provided in this application embodiment can be executed by terminal 110, or by terminal 110 and server 120 in cooperation. In the case where it is executed by terminal 110, terminal 110 can provide a sentiment analysis interface. After the user inputs multimodal data, terminal 110 can obtain the sentiment analysis results of the aspects according to the sentiment analysis method of the first aspect described above and display them on the sentiment analysis interface. In the case where terminal 110 and server 120 cooperate to execute the method, terminal 110 can provide a sentiment analysis interface. After the user inputs multimodal data, terminal 110 can transmit the multimodal data to server 120. Server 120 can obtain the sentiment analysis results of the aspects according to the sentiment analysis method of the first aspect described above and return them to terminal 110 to display them on the sentiment analysis interface. The sentiment analysis method provided in this application embodiment can be applied to sentiment analysis systems including multimodal fine-grained e-commerce review sentiment analysis systems, restaurant review sentiment analysis systems, and microblog sentiment analysis systems.
[0057] The following sections will be based on, for example Figure 1 The application environment shown, together with the various embodiments and corresponding figures, further illustrates the sentiment analysis method of this application.
[0058] In one embodiment, such as Figure 2 As shown, a sentiment analysis method is provided, which can be executed by server 120 or terminal 110, and the method may include the following steps:
[0059] Step S201: Obtain the multimodal data to be analyzed.
[0060] The fine-grained sentiment analysis is based on multimodal data, which refers to the integration or fusion of multiple types of data. In this step, the multimodal data includes text content, non-text content, and aspects within the text content. The text content can be a piece of text; the non-text content can be images, audio, video, or other content of different text types. This application will primarily use images as an example of non-text content. The aspects within the text content can be some or all of the aforementioned aspects. As an illustrative example, the text content could be: "The wolf, pretending to be very kind, says, 'Little lamb, I know I was wrong. Don't be afraid, come out quickly, I promise I won't eat you.'" The non-text content could be an image containing images of wolves, grass, etc., and the aspect within the text content could be: "Little lamb."
[0061] Step S202: Based on the non-text content, obtain object-related information including object text tags and their associated knowledge.
[0062] In this step, for non-text content in multimodal data, text tags corresponding to objects contained in the non-text content can be identified. The text tags corresponding to objects contained in the non-text content can be denoted as object text tags. The specific form of the object text tag can be the name of the object contained in the non-text content. Then, the associated knowledge can be obtained based on the object text tag. In this case, phrases associated with the object text tag can be searched to obtain the associated knowledge of the object text tag. The phrases associated with the object text tag can be synonyms, near-synonyms, and semantically related words, thereby obtaining the associated knowledge of the object text tag. Then, object-related information containing the object text tag and its associated knowledge is obtained.
[0063] Furthermore, in one embodiment, step S202, obtaining object-related information including object text tags and their associated knowledge based on non-text content, may include:
[0064] Input non-text content into the trained object recognition model to obtain the object text labels output by the object recognition model; obtain the associated knowledge of the object text labels based on the established knowledge base; and obtain object-related information based on the object text labels and their associated knowledge.
[0065] This embodiment relates to the processing of non-textual content in multimodal data, which can be divided into an object detection and recognition stage and an external knowledge retrieval stage, thereby obtaining object-related information including object text tags and their associated knowledge. Specifically, in the object detection and recognition stage, a trained object recognition model can identify the text tags corresponding to objects contained in the non-textual content; that is, the trained object recognition model identifies the object text tags. As mentioned above, the specific form of the object text tag can be the name of the object contained in the non-textual content. Taking an image as an example of non-textual content, combined with... Figure 3 The image is input into a trained object recognition model, which identifies the objects contained in the image and outputs the corresponding text labels for each object, including "wolf" and "grass." It should be noted that current sentiment analysis solutions directly use relevant models to encode images and text, and fuse multimodal features through attention mechanisms. However, the differences in modal representation spaces hinder fusion, thus affecting the performance of sentiment analysis. Therefore, this embodiment indirectly uses modal information from non-textual content such as images. Specifically, the text labels corresponding to the objects contained in the image can be obtained through an object recognition model (such as Faster-RCNN), and used as a substitute input for the image modality. This allows the two modal features to reside in the same vector space, improving the fusion effect of image and text modalities and making modality fusion more effective. The next stage is the external knowledge retrieval stage. In this stage, the associated knowledge of the object text tags is obtained based on the established knowledge base. Specifically, the object text tags can be used to search for related phrases in the knowledge base to obtain associated knowledge. As mentioned earlier, these associated phrases can be synonyms, near-synonyms, and semantically related words, thus yielding the associated knowledge of the object text tags. After obtaining the object text tags and their associated knowledge, object-related information containing the object text tags and their associated knowledge can be obtained.
[0066] Specifically, in some embodiments, obtaining the associated knowledge of object text tags based on the established knowledge base in the above embodiments may include:
[0067] Based on the object text tags, determine the associated text tags and relationships of the objects in the established knowledge base; based on the associated text tags and relationships of the objects, obtain the associated knowledge of the object text tags.
[0068] In this embodiment, based on the object text tags, associated text tags and their relationships can be found in knowledge bases such as the DBpedia external knowledge graph. Object-related text tags refer to text tags in an established knowledge base that are associated with the object text tags. Relationships can include the object text tags and the relationships between them. Then, the object-related text tags and their relationships can be used as the associated knowledge of the object text tags. For example, such as... Figure 4 As shown, based on the object text tag "wolf," associated text tags such as "carnivore," "sheep," "dog," and "sheepdog" and their relationships can be obtained from the knowledge base. This expands the text tags, not only compensating for potentially lost information in the image but also introducing external knowledge. It should be noted that replacing images with object text tags may result in the loss of important information. Therefore, step S203 of this application uses the object text tags as indexes to search for related synonyms, near-synonyms, and semantically related words in external knowledge bases such as DBpedia, as associated knowledge for the object text tags. This achieves the effect of introducing external knowledge into multimodal fine-grained sentiment analysis, further supplementing and enriching image modal information, and filtering noise in the image information.
[0069] Step S203: Using aspect items as connection nodes, connect text content and object-related information to construct a multimodal semantic graph containing nodes corresponding to aspect items, text content, and object-related information.
[0070] Step S204: Obtain the initial feature representations corresponding to the aspect items, text content, and object-related information.
[0071] Steps S203 and S204 above mainly involve constructing a multimodal semantic graph and determining the initial feature representations of each node in the multimodal semantic graph. Specifically, unlike current technologies that use sequence patterns to interact with text and image modal representations, this application uses graph patterns to interact with information from both modalities. To ensure that the multimodal semantic graph contains nodes corresponding to aspects, text content, and object-related information, in step S203, aspects are used as connecting nodes to connect text content and object-related information (including object text tags and their associated knowledge), thereby constructing a multimodal semantic graph containing nodes corresponding to aspects, text content, and object-related information (including object text tags and their associated knowledge).
[0072] In some embodiments, such as Figure 5 As shown, step S203, which uses aspect items as connecting nodes to connect text content and object-related information, constructs a multimodal semantic graph containing nodes corresponding to aspect items, text content, and object-related information. Specifically, this may include:
[0073] Step S501: Determine the first sub-graph corresponding to the text content.
[0074] In this step, a subgraph refers to a part of the multimodal semantic graph to be formed. The first subgraph corresponds to the text content, and the first subgraph may contain the words and their relationships in the text content.
[0075] Step S502: Based on the association relationships in the association knowledge, obtain a second subgraph containing the nodes corresponding to the object text labels and their respective association knowledge.
[0076] In this step, the second subgraph can correspond to object-related information containing object text labels and their associated knowledge. Specifically, the second subgraph can contain the nodes corresponding to the object text labels and their associated knowledge. The second subgraph containing the nodes corresponding to the object text labels and their associated knowledge can be formed based on the object text labels and the object-related text labels and their relationships in the associated knowledge.
[0077] Step S503: Using aspect terms as connection nodes between the first subgraph and the second subgraph, connect the first subgraph and the second subgraph to obtain a multimodal semantic graph.
[0078] This step is in the graph construction stage. Aspect terms are used as connection nodes between the first subgraph and the second subgraph to connect the first subgraph and the second subgraph, thereby obtaining a multimodal semantic graph.
[0079] In a specific embodiment, step S501 may include: obtaining a dependency parsing tree of the text content to obtain a first subgraph. Step S502 may include: aligning the connections between nodes corresponding to object text tags and nodes corresponding to their associated knowledge with the association relationships in the associated knowledge to obtain a second subgraph. Step S503 may include: replacing the root node of the first subgraph with the node corresponding to the aspect item, and connecting the node corresponding to the aspect item with the node corresponding to the object text tag in the second subgraph to obtain a multimodal semantic graph.
[0080] In this embodiment, for the first subgraph, the dependency parsing tree of the text content is specifically obtained to obtain the first subgraph. The Spacy tool can be used to analyze the text content, and the resulting dependency parsing tree is used as the first subgraph. For the second subgraph, the association knowledge can include the associated text tags and relationships of each object. Therefore, when forming the second subgraph, the connections between the nodes corresponding to the object text tags and the nodes corresponding to their associated knowledge (including the nodes corresponding to the associated text tags of each object) can correspond to the relationships in the association knowledge, thus obtaining the second subgraph and maintaining the connections of these tags in the knowledge base. For the connection between the first and second subgraphs, the root node of the first subgraph can be replaced with the node corresponding to the aspect item, and then the node corresponding to the aspect item can be connected to the node corresponding to each object text tag in the second subgraph, thereby constructing a multimodal semantic graph. An exemplary multimodal semantic graph formed through this embodiment is shown below. Figure 6 As shown, the edges connecting nodes in this multimodal semantic graph can have the same weight.
[0081] In step S204, the main task is to obtain the initial feature representations corresponding to the aspect items, text content, and object-related information, so that each node in the multimodal semantic graph has its own corresponding initial feature representation. In specific implementation, this initial feature representation can use embedding feature representation, which can be obtained through neural network models capable of processing sequences, such as BERT, RNN, and CNN.
[0082] In some embodiments, the initial feature representations corresponding to the aspect items, text content, and object-related information obtained in step S204 specifically include:
[0083] For text content, the contextual representation of the text content is obtained through a pre-trained language model, resulting in the initial feature representation of the text content. For aspect items and object-related information, the embedding feature representations corresponding to each aspect item and object-related information are obtained through the embedding mapping function of the pre-trained language model. The embedding features corresponding to each aspect item and object-related information are then transformed into the feature space of the contextual representation through a linear layer mapping function, resulting in the initial feature representations corresponding to each aspect item and object-related information.
[0084] In this embodiment, for text content, the pre-trained language model BERT can be used as a sequence encoder to obtain the initial feature representation of the text content. For a given text content x = {x1, x2, ..., xn}, where n is the maximum sequence number of the text content, the pre-trained language model BERT is used to obtain the context representation h = {h1, h2, ..., hn} of the text content, and this context representation is used as the initial feature representation of the text content. For aspect terms, object text labels, and object-related text labels in their associated knowledge, all are in phrase form. In this embodiment, the embedding mapping function of the pre-trained language model BERT is directly used to obtain the embedding feature representations corresponding to each aspect term, object text label, and their associated knowledge, denoted as e. i 'i' represents the index in which the aspect, object text label, and object-related text label are distinguished. Then, using the same linear layer mapping function Linear(*), the embedding feature representations of these phrases are transformed onto the feature space of the aforementioned context representation, thereby obtaining the initial feature representation h corresponding to each aspect, object text label, and their associated knowledge. i =Linear(e i ).
[0085] Thus, we can obtain a multimodal semantic graph containing the nodes corresponding to aspect terms, text content, and object-related information (including object text labels and their associated knowledge), as well as the initial feature representation of each node in the multimodal semantic graph.
[0086] Step S205: Input the multimodal semantic graph and the initial feature representation into the trained graph convolutional neural network to obtain the target feature representation corresponding to the aspect terms output by the graph convolutional neural network.
[0087] This step primarily uses a trained Graph Convolutional Neural Network (GCN) to encode the multimodal semantic graph by combining the initial feature representations of each node, fusing and interacting information from multiple modalities. The GCN can obtain the feature representation of each node; this step obtains the target feature representation of the node corresponding to the aspect terms output by the GCN. Specifically, given the multimodal semantic graph and the initial feature representations of each node, the GCN(*) is used to fuse and interact the initial feature representations of each node, resulting in the output O of the GCN.
[0088] O = GCN(g,K)
[0089] Where g represents the multimodal semantic graph, and K represents the initial feature representation of each node (i.e., the contextual representation of the text content and the feature representation of each word group after linear transformation).
[0090] Step S206: Based on the target feature representation, obtain the sentiment analysis results of the aspect items.
[0091] In this step, the sentiment analysis results of the aspect items are obtained based on the target feature representation of the nodes corresponding to the aspect items output by the graph convolutional neural network. That is, this application ultimately uses the target feature representation of the nodes corresponding to the aspect items to obtain the sentiment polarity of the aspect items.
[0092] In some embodiments, step S206, obtaining the sentiment analysis result of the aspect items based on the target feature representation, specifically includes:
[0093] The target feature representation output by the graph convolutional neural network is input into a trained sentiment classifier; based on the sentiment classification results of the aspect terms output by the sentiment classifier, the sentiment analysis results of the aspect terms are obtained.
[0094] In this embodiment, the target features of the nodes corresponding to the aspect terms output by the graph convolutional neural network can be represented. The input is fed into a trained sentiment classifier, which outputs a corresponding sentiment classification result based on the target feature representation. This can include sentiment classification results such as positive, neutral, or negative. This sentiment classification result can then be used as the sentiment analysis result of the aspect items.
[0095] Overall, such as Figure 7 As shown, when utilizing non-textual modal content such as images, this application can use an object recognition model to complete object recognition, and then use object text labels to fuse text modal features, so that the features of the two modalities are in the same vector space, improving the fusion effect of non-textual modalities such as image modalities and text modalities. Before fusing the two modal information, in order to complete the information of non-textual modalities, object text labels can be used to search for related knowledge such as synonyms, near-synonyms and semantically related words in an external knowledge base. This can not only enrich the information of non-textual modalities such as image modalities, but also filter out noise in the information of non-textual modalities such as image modalities. In order to fuse the two modal information more effectively, aspect terms are used as a bridge to connect text modalities and image modalities and related knowledge from the outside, and a multimodal semantic graph is constructed. Then, a graph convolutional neural network is used to encode each node to fuse its own initial feature representation (initial feature representation 1, initial feature representation 2 and initial feature representation 3). Finally, the target feature representation of the node corresponding to the aspect term is used to classify by a sentiment classifier to obtain the sentiment classification result and obtain the sentiment analysis result of the aspect term.
[0096] The sentiment analysis method described in the above embodiments acquires multimodal data containing text content, non-text content, and aspects within the text content. Based on the non-text content, it obtains object-related information, including object text tags and their associated knowledge. Aspects are used as connection nodes to link the text content and object-related information, thereby constructing a multimodal semantic graph. Initial feature representations corresponding to each aspect, text content, and object-related information are obtained. The multimodal semantic graph and initial feature representations are input into a trained graph convolutional neural network (GCNN). The target feature representations corresponding to the aspect items output by the GCNN are obtained, and the sentiment analysis results for the aspect items are obtained based on these target feature representations. This scheme can find associated knowledge of object text tags, enrich the modal information of non-text content, use aspect items to connect text content and object text tags and their associated knowledge to construct a multimodal semantic graph, and employ a graph convolutional neural network to fuse feature representations. This allows all modalities to be in the same space, improving the multimodal fusion effect. Furthermore, by fusing multimodal features on a graph structure containing associated knowledge, it improves the accuracy of fine-grained sentiment analysis.
[0097] In one embodiment, a graph convolutional neural network can be trained through the following steps, such as... Figure 8 As shown, it specifically includes:
[0098] Step S801: Obtain training samples and corresponding labeled sentiment tags.
[0099] In this step, the training samples include multimodal semantic map samples and corresponding initial feature representation samples. Multimodal data can be collected as multimodal data samples. At the same time, during training, it is necessary to determine the sentiment label corresponding to the multimodal data sample. The sentiment label can be a sentiment label labeled manually. Then, the corresponding multimodal semantic map can be formed based on the multimodal data sample as the multimodal semantic map sample, and the corresponding initial feature representation can be obtained as the initial feature representation sample.
[0100] Step S802: Input the multimodal semantic graph samples and the corresponding initial feature representation samples into the graph convolutional neural network to be trained, and output the predicted target feature representation corresponding to the aspect terms from the graph convolutional neural network to be trained.
[0101] Step S803: Train the graph convolutional neural network to be trained based on the predicted target feature representation and the real sentiment label to obtain the trained graph convolutional neural network.
[0102] In steps S802 and S803, multimodal semantic graph samples and corresponding initial feature representation samples are input into the graph convolutional neural network (GCNN) to be trained. The GCNN can output the predicted target feature representation corresponding to the aspect node based on the multimodal semantic graph samples and corresponding initial feature representation samples. That is, the target feature representation belongs to the prediction result of the GCNN to be trained. Then, the GCNN to be trained is trained based on the predicted target feature representation and the real sentiment label. When the preset training completion conditions are met, the trained GCNN can be obtained. The preset training completion conditions may be reaching a certain number of iterations, etc. In practical applications, when training the GCNN, the sentiment classifier can output the predicted sentiment label based on the predicted target feature representation. Then, the predicted sentiment label is used to fit the real sentiment label. Thus, the network parameters of the GCNN can be continuously adjusted based on the difference between the two and the gradient descent optimization algorithm until the difference between the two meets the preset difference condition, and the trained GCNN is obtained.
[0103] In one embodiment, such as Figure 9 As shown, a sentiment analysis method is also provided, which can be executed by terminal 110 or by terminal 110 and server 120 in cooperation. The method may include the following steps:
[0104] Step S901: Provide a sentiment analysis interface.
[0105] Step S902: Receive multimodal data input by the user in the multimodal data input area.
[0106] like Figure 10 As shown, terminal 110 provides a sentiment analysis interface, which may include a multimodal data input area and a sentiment analysis result display area. The multimodal data input area allows users to input multimodal data, including text content, non-text content, and aspect items. The sentiment analysis result display area outputs and displays the sentiment analysis results for each aspect item. After receiving the multimodal data input by the user in the multimodal data input area, terminal 110 can directly obtain the sentiment analysis results for each aspect item using the sentiment analysis method described in any of the previous embodiments, or terminal 110 can send the multimodal data to server 120, where server 120 obtains the sentiment analysis results for each aspect item using the sentiment analysis method described in any of the previous embodiments and returns them to terminal 110. Terminal 110 then receives the sentiment analysis results returned by server 120.
[0107] Step S9031: When the multimodal data input by the user also includes aspect items, the sentiment analysis results of the aspect items are displayed in the sentiment analysis results display area.
[0108] Step S9032: When the multimodal data input by the user does not contain aspect items, the sentiment analysis results of each aspect item contained in the text content are displayed in the sentiment analysis results display area.
[0109] For steps S9031 and S9032, as mentioned above, the multimodal data input by the user may or may not contain aspect items. If the multimodal data input by the user also contains aspect items, in step S9031, the terminal 110 displays the sentiment analysis result of that aspect item in the sentiment analysis result display area, so that the user can obtain the sentiment analysis result of the aspect item they need to know. If the multimodal data input by the user does not contain aspect items, in step S9032, the terminal 110 displays the sentiment analysis results of all aspects of the text content in the sentiment analysis result display area. That is, the user does not need to specify specific aspect items. At this time, the sentiment analysis results of all aspects of the text content are identified for the user to understand the sentiment analysis results of all aspects of the text content.
[0110] This embodiment provides users with a convenient and flexible way to obtain fine-grained sentiment analysis results based on multimodal data while ensuring the accuracy of fine-grained sentiment analysis.
[0111] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0112] In one embodiment, a computer device, which may be a server, is provided, and its internal structure is shown in Figure 11(a). The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores multimodal data and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a sentiment analysis method.
[0113] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in Figure 11(b). The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a sentiment analysis method. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0114] Those skilled in the art will understand that the structures shown in Figures 11(a) and 11(b) are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.
[0115] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0116] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.
[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0120] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A sentiment analysis method, characterized in that, The method includes: Acquire multimodal data to be analyzed; the multimodal data includes text content, non-text content, and aspects of the text content; the non-text content is images; Based on the non-text content, object-related information, including object text tags and their associated knowledge, is obtained; the object text tags are the text tags corresponding to the objects contained in the non-text content; the associated knowledge includes phrases associated with the object text tags. Obtain the dependency parsing tree of the text content to get the first subgraph; make the connection between the node corresponding to the object text label and the node corresponding to the associated knowledge correspond to the association relationship in the associated knowledge to get the second subgraph; replace the root node of the first subgraph with the node corresponding to the aspect item, and connect the node corresponding to the aspect item with the node corresponding to the object text label in the second subgraph to get the multimodal semantic graph. Obtain the initial feature representations corresponding to each of the aspect items, the text content, and the object-related information; The multimodal semantic graph and the initial feature representation are input into a trained graph convolutional neural network to obtain the target feature representation corresponding to the aspect terms output by the graph convolutional neural network. Based on the target feature representation, the sentiment analysis results of the aspect items are obtained.
2. The method according to claim 1, characterized in that, The step of obtaining object-related information, including object text tags and their associated knowledge, based on the non-text content includes: Input the non-text content into a trained object recognition model and obtain the object text labels output by the object recognition model; Based on the established knowledge base, obtain the associated knowledge of the object's text tags; Based on the object's text tags and their associated knowledge, relevant information about the object is obtained.
3. The method according to claim 2, characterized in that, The step of obtaining the associated knowledge of the object text tags based on the established knowledge base includes: Based on the object text tags, determine the object-related text tags and their relationships in the established knowledge base; the object-related text tags are the text tags in the established knowledge base that are associated with the object text tags. Based on the object's associated text tags and their relationships, the association knowledge of the object's text tags is obtained.
4. The method according to claim 1, characterized in that, The process of obtaining the initial feature representations corresponding to each of the aspect items, the text content, and the object-related information includes: For the text content, the contextual representation of the text content is obtained through a pre-trained language model, thus obtaining the initial feature representation of the text content; For the aspect items and object-related information, the embedding feature representations corresponding to each aspect item and object-related information are obtained through the embedding mapping function of the pre-trained language model. The embedding features corresponding to each aspect item and object-related information are transformed into the feature space where the context representation is located through the linear layer mapping function, so as to obtain the initial feature representations corresponding to each aspect item and object-related information.
5. The method according to claim 1, characterized in that, The step of obtaining the sentiment analysis result of the aspect item based on the target feature representation includes: The target feature representation output by the graph convolutional neural network is input into the trained sentiment classifier; Based on the sentiment classification results of the aspect items output by the sentiment classifier, the sentiment analysis results of the aspect items are obtained.
6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain training samples and corresponding labeled sentiment tags; the training samples include multimodal semantic graph samples and corresponding initial feature representation samples; The multimodal semantic graph samples and their corresponding initial feature representation samples are input into the graph convolutional neural network to be trained, and the graph convolutional neural network to be trained outputs the predicted target feature representations corresponding to the aspect terms. Based on the predicted target feature representation and the corresponding sentiment labels of the training samples, the graph convolutional neural network to be trained is trained to obtain the trained graph convolutional neural network.
7. A sentiment analysis method, characterized in that, The method includes: A sentiment analysis interface is provided; the sentiment analysis interface includes a multimodal data input area and a sentiment analysis result display area; The system receives multimodal data input by a user in the multimodal data input area; wherein the multimodal data input by the user includes text content and non-text content; the non-text content is an image; The sentiment analysis results display area shows the sentiment analysis results of the aspects contained in the multimodal data to be analyzed; wherein, when the multimodal data input by the user also contains aspects, the aspects contained in the multimodal data to be analyzed are the aspects contained in the multimodal data input by the user; when the multimodal data input by the user does not contain aspects, the aspects contained in the multimodal data to be analyzed are the various aspects contained in the text content; The sentiment analysis results are obtained by analyzing the multimodal data to be analyzed according to the method described in any one of claims 1 to 6.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6 or claim 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6 or claim 7.