Chinese sentiment classification method and device based on multiple modes and electronic equipment

Abstract

The invention discloses a Chinese sentiment classification method and device based on multiple modes and electronic equipment. The method comprises the steps of obtaining a target Chinese text; determining attribute words of the target Chinese text, and constructing a dependency font graph of the attribute words based on the attribute words; vectorizing the target Chinese text and the dependency font graph to obtain a text vector matrix and an image vector matrix; respectively determining attention scores of the text vector matrix and the image vector matrix; the emotion polarity of the target Chinese text is determined based on the attention score, the text vectors in the text vector matrix and the image vectors in the image vector matrix, and the emotion polarity comprises positive emotion or negative emotion. The Chinese emotion prediction method and device solve the technical problems that an existing Chinese emotion analysis method is single in analysis dimension, character pattern information of Chinese characters is not fully fused, only text information is considered, and consequently Chinese emotion prediction accuracy is low.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and more specifically, to a multimodal Chinese sentiment classification method, device and electronic device. Background Art

[0002] Sentiment classification, as a basic task in the field of natural language processing, aims to automatically identify the emotional polarity expressed in texts. Among them, attribute-level sentiment classification focuses on identifying positive and negative emotions related to specific attributes in texts. Early research mainly relied on machine learning methods, such as support vector machines (SVMs), to extract sentiment information through feature engineering and sentiment dictionaries. However, with the continuous development of deep learning technology, neural network models that can automatically extract features, such as recurrent neural networks (RNNs) and convolutional neural networks (CNNs), have gradually replaced traditional machine learning methods due to their advantages in effectiveness and efficiency.

[0003] In recent years, deep learning-based models have been increasingly used in the field of attribute-level sentiment classification, from the initial target-dependent long short-term memory network (TD-LSTM) and target-context long short-term memory network (TC-LSTM), which use long short-term memory networks (LSTM) to extract the contextual information of attribute words respectively, to the attention-based LSTM with Aspect Embedding (ATAE-LSTM) model that integrates attribute information. By concatenating attribute word vectors in the input layer and the hidden layer of LSTM and introducing the attention mechanism, the classification effect has been significantly improved. The gated convolutional neural network (GCAE) model for attribute-level sentiment analysis further combines CNN and the gating mechanism to determine the sentiment tendency based on given attribute words. However, most of the above methods are based on English datasets. When performing sentiment analysis on Chinese, they omit the information of Chinese glyphs. Most of them are from the perspective of text and do not focus on using glyph information to perform attribute-level sentiment classification of Chinese. Most of them focus on capturing the internal structure and ignore the importance of external relationships between Chinese character sequences.

[0004] Therefore, the existing Chinese sentiment analysis methods have a single analysis dimension, do not fully integrate the glyph information of Chinese characters, and only consider text information, resulting in a low accuracy rate in Chinese sentiment prediction.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a Chinese sentiment classification method, device and electronic device based on multimodality, so as to at least solve the technical problems that the existing Chinese sentiment analysis method has a single analysis dimension, does not fully integrate the glyph information of Chinese characters, only considers text information, and leads to a low accuracy rate in Chinese sentiment prediction.

[0007] According to one aspect of an embodiment of the present application, a multimodal Chinese sentiment classification method is provided, comprising: obtaining a target Chinese text; determining the attribute words of the target Chinese text, and constructing a dependency glyph graph of the attribute words based on the attribute words, wherein the dependency glyph graph contains the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; vectorizing the target Chinese text and the dependency glyph graph respectively to obtain a text vector matrix and an image vector matrix; determining the attention scores of the text vector matrix and the image vector matrix respectively; determining the sentiment polarity of the target Chinese text based on the attention scores, the text vectors in the text vector matrix, and the image vectors in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

[0008] In some embodiments of the present application, determining the attribute words of the target Chinese text includes: segmenting the target Chinese text and marking the part of speech; based on the part of speech, identifying words in the target Chinese text that belong to a preset part of speech as candidate attribute words, wherein the preset part of speech is a noun; screening the candidate attribute words based on the context to determine the attribute words of the target Chinese text.

[0009] In some embodiments of the present application, a dependency glyph graph of attribute words is constructed based on attribute words, including: parsing the grammatical structure of the target Chinese text to obtain the grammatical dependency relationships of each word in the target Chinese text; determining a dependency tree based on the attribute words and the grammatical dependency relationships: taking the attribute words as the root node of the dependency tree, identifying words directly connected to the attribute words based on the grammatical dependency relationships as first-level child nodes, and the first-level child nodes are used to reflect direct grammatical associations; traversing the dependency relationships, adding all words that have grammatical dependency relationships with the attribute words as child nodes to obtain a dependency tree; embedding the glyph image information of each Chinese character into the dependency tree to obtain a dependency glyph graph, wherein the glyph image information is used to display the visual shape representation of Chinese characters in one or more fonts.

[0010] In some embodiments of the present application, the target Chinese text and the dependency glyph graph are respectively vectorized to obtain a text vector matrix and an image vector matrix, including: vectorizing the target Chinese text to obtain a text vector matrix: performing word embedding processing on each word in the target Chinese text: vectorizing each word in the target Chinese text to obtain a first vector, wherein the first vector is used to indicate the semantic information of each word and the usage method in the context; performing position embedding processing on each word in the target Chinese text: vectorizing the position of each word in the target Chinese text to obtain a second vector, wherein the second vector is used to indicate the position order and position weight of each word in the target Chinese text; performing paragraph embedding processing on the target Chinese text: vectorizing each sentence in the target Chinese text to obtain a third vector, wherein the third vector is used to assign a unique paragraph identifier to each sentence; and obtaining a text vector matrix based on the first vector, the second vector, and the third vector.

[0011] In some embodiments of the present application, when paragraph embedding processing is performed on the target Chinese text, the sentences in the target Chinese text are determined in the following manner: the target Chinese text is divided according to attribute words, the text on the left side of the attribute word is taken as the first sentence, and the text on the right side of the attribute word is taken as the second sentence.

[0012] In some embodiments of the present application, the target Chinese text and the dependency glyph graph are respectively vectorized to obtain a text vector matrix and an image vector matrix, including: vectorizing the dependency glyph graph to obtain an image vector matrix; dividing the dependency glyph graph into multiple image regions containing glyph images of a single Chinese character, wherein each image region corresponds to the glyph information of a Chinese character; encoding each image region to obtain a vector representation of each image region; splicing the vector representations of each image region to obtain an initial image vector matrix; processing the initial image vector matrix based on a preset linear layer to obtain an image vector matrix, wherein the linear layer is used to adjust the dimension and feature representation of the initial image vector matrix.

[0013] In some embodiments of the present application, the attention scores of the text vector matrix and the image vector matrix are determined respectively: the image vectors corresponding to the text vectors in the text vector matrix in the image vector matrix are determined to obtain text-image pairs; the similarity between the text vector and the image vector in each text-image pair is calculated; based on the similarity, attention weights are assigned to the text vectors in the text vector matrix and the corresponding image vectors in the image vector matrix; and the text attention scores and the image attention scores are determined based on the attention weights.

[0014] In some embodiments of the present application, the sentiment polarity of the target Chinese text is determined based on the attention score, the text vectors in the text vector matrix, and the image vectors in the image vector matrix, including: weighting the text vector matrix and the image vector matrix according to the attention score to obtain multimodal representation features; inputting the multimodal representation features into the decision layer to obtain the sentiment polarity score, wherein the decision layer is used to convert the multimodal representation features into a score or probability representing the sentiment polarity; when the sentiment polarity score is greater than or equal to a preset score threshold, determining that the sentiment of the target Chinese text is positive, and when the sentiment polarity score is less than the preset score threshold, determining that the sentiment of the target Chinese text is negative.

[0015] According to another aspect of an embodiment of the present application, a multimodal Chinese sentiment classification device is also provided, including: an acquisition module for acquiring a target Chinese text; a first determination module for determining the attribute words of the target Chinese text, and constructing a dependency glyph graph of the attribute words based on the attribute words, wherein the dependency glyph graph contains the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; a vectorization module for vectorizing the target Chinese text and the dependency glyph graph respectively to obtain a text vector matrix and an image vector matrix; a second determination module for determining the attention scores of the text vector matrix and the image vector matrix respectively; a third determination module for determining the sentiment polarity of the target Chinese text based on the attention score, the text vectors in the text vector matrix, and the image vectors in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

[0016] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned multimodal-based Chinese sentiment classification method.

[0017] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above-mentioned multimodal-based Chinese sentiment classification method is executed when the program is run.

[0018] According to another aspect of an embodiment of the present application, a computer program product is also provided, including computer instructions, which implement the above-mentioned multimodal-based Chinese sentiment classification method when executed by a processor.

[0019] In an embodiment of the present application, a target Chinese text is obtained; attribute words of the target Chinese text are determined, and a dependency glyph graph of the attribute words is constructed based on the attribute words, wherein the dependency glyph graph contains glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; the target Chinese text and the dependency glyph graph are vectorized respectively to obtain a text vector matrix and an image vector matrix; attention scores of the text vector matrix and the image vector matrix are determined respectively; the sentiment polarity of the target Chinese text is determined based on the attention scores, the text vectors in the text vector matrix, and the image vectors in the image vector matrix. Among them, sentiment polarity includes positive sentiment or negative sentiment. By determining the attribute words of the target Chinese text, constructing a dependency glyph graph containing the glyph image information of Chinese characters and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text, multimodal information is integrated. The final sentiment polarity is determined based on the attention score, the text vector in the text vector matrix, and the image vector in the image vector matrix. The purpose of predicting sentiment polarity based on multimodal information is achieved, thereby solving the technical problem that the existing Chinese sentiment analysis method has a single analysis dimension, does not fully integrate the glyph information of Chinese characters, and only considers text information, resulting in a low accuracy rate in Chinese sentiment prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a multimodal Chinese sentiment classification method provided in an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a multimodal Chinese sentiment classification method provided according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of the structure of a multimodal language model provided according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of the structure of a dependency glyph graph provided according to an embodiment of the present application;

[0025] Figure 5 is a structural diagram of a multimodal interaction model provided according to an embodiment of the present application;

[0026] Figure 6 It is a structural schematic diagram of a multimodal Chinese sentiment classification device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0028] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0031] Attribute-level sentiment classification: A fine-grained sentiment classification task that determines the sentiment tendency of different attribute words in the text.

[0032] Dependency relationship: The dependency relationship is represented by core words and dependent words, and each core word corresponds to the center of its components.

[0033] Multimodality: Multimodality refers to the use of two or more senses to exchange information simultaneously.

[0034] In the related art, the existing Chinese sentiment analysis method has a single analysis dimension, does not fully integrate the Chinese character shape information, and only considers the text information, resulting in a low accuracy rate of Chinese sentiment prediction. In order to solve this problem, the present application provides a relevant solution in the embodiment, which is described in detail below.

[0035] According to an embodiment of the present application, a method embodiment of a Chinese sentiment classification method based on multimodality is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a multimodal Chinese sentiment classification method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multimodal Chinese sentiment classification method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned multimodal Chinese sentiment classification method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0039] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0041] In the above operating environment, the embodiment of the present application provides a flowchart of a Chinese sentiment classification method based on multimodality, such as Figure 2 As shown, the method comprises the following steps:

[0042] Step S202, obtaining target Chinese text. The target Chinese text may include but is not limited to comment information on the target object (such as a product), and the comment information may be extracted from a web page or input by a user. It is understandable that the method for obtaining the target Chinese text is not limited to the above-mentioned method.

[0043] Step S204, determining the attribute words of the target Chinese text, and constructing a dependency glyph graph of the attribute words based on the attribute words, wherein the dependency glyph graph contains the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text. This method can accurately identify key sentiment-bearing words in the text, and is particularly effective for sentiment analysis of texts such as news titles and product descriptions, because it can capture the core features of the described object.

[0044] In the technical solution provided in step S204, there are multiple ways to determine the attribute words of the target Chinese text, for example: segmenting the target Chinese text and marking the part of speech; based on the part of speech, identifying the words in the target Chinese text that belong to the preset part of speech as candidate attribute words, where the preset part of speech is noun; screening the candidate attribute words based on the context to determine the attribute words of the target Chinese text.

[0045] This method not only takes into account the grammatical structure of the text, but also utilizes the visual information of Chinese characters, providing a new perspective for text sentiment analysis in areas such as calligraphy works and art design.

[0046] The following is a specific embodiment:

[0047] For example, the target Chinese text is "The dessert tastes delicious". After the text is segmented, the noun "dessert" is obtained as a candidate attribute word. The following is "tastes delicious", and it can be concluded that "dessert" is the attribute word.

[0048] There are many methods for constructing a dependency glyph graph of attribute words based on attribute words. For example, the following methods can be used: parse the grammatical structure of the target Chinese text to obtain the grammatical dependency relationship of each word in the target Chinese text; determine the dependency tree based on the attribute words and the grammatical dependency relationship: take the attribute word as the root node of the dependency tree, identify the words directly connected to the attribute word based on the grammatical dependency relationship as the first-level child nodes, and the first-level child nodes are used to reflect the direct grammatical association; traverse the dependency relationship, add all words that have grammatical dependency relationships with the attribute word as child nodes, and obtain a dependency tree; embed the glyph image information of each Chinese character into the dependency tree to obtain a dependency glyph graph, wherein the glyph image information is used to display the visual shape expression of Chinese characters in one or more fonts.

[0049] The following is a specific embodiment:

[0050] like Figure 4As shown, a structural schematic diagram of a dependency graph is provided, and the target Chinese text is "The packaging of the mask is very good and very cute." The grammatical structure of the target Chinese text is parsed (for example, using a Language Technology Platform Parser (LTP parser)) to obtain the grammatical dependency relationship between the words "mask, packaging, very good, very cute": "mask" and "packaging" are compound word relationships, "packaging" and "very good" are noun subject relationships, and "very good" and "very cute" are parallel relationships, such as Figure 4 , compound (compound relationship): In dependency grammar, compound represents the composition relationship of a compound word, in which a certain word is the core part of the compound word, and another word is attached to the core word, usually used to describe or refine the meaning of the core word. For example, "dining table" can be decomposed into "table" as the core word and "meal" as the additional word, which means "a table for eating". nsubj (noun subject relationship): nsubj refers to the noun subject, which is the main agent of the verb. For example, in the sentence "he eats", "he" is the noun subject of the verb "eats", indicating the performer of the action. conj (parallel relationship): conj represents a parallel relationship, that is, the relationship between two or more parallel words or sentences. Parallel words are usually connected by conjunctions (such as "and" and "or"), and they play the same role in the sentence. For example, in the sentence "Xiao Ming and Xiao Hong go to school", "Xiao Ming" and "Xiao Hong" are parallel subjects, connected by "and", and they jointly perform the action of "going to school".

[0051] Take the attribute word mask as the root node of the dependency tree, take the package as the first-level child node according to the grammatical dependency relationship, continue to traverse the dependency relationship, add "very good" and "very cute" as child nodes, and get the dependency tree. Then embed the glyph image information of each Chinese character into the corresponding node position in the dependency tree to get the dependency glyph graph, such as Figure 4 As shown in Figure 2. This dependency graph has at least two advantages: first, each attribute word has its own dependency tree and is less affected by irrelevant node relationships; second, encoding the tree structure based on the image is simpler than using a graph neural network to encode dependencies. First, the target attribute word is placed on the root, and then the nodes directly connected to the attribute word are set as child nodes and their original dependencies are retained. Finally, other irrelevant dependencies are discarded. If the target Chinese text contains multiple attribute words, a unique dependency glyph graph is constructed for each attribute word.

[0052] Step S206, vectorizing the target Chinese text and the dependent glyph graph respectively to obtain a text vector matrix and an image vector matrix.

[0053] In the technical solution provided in step S206, there are many ways to implement vectorization of the target Chinese text and the dependent glyph graph to obtain a text vector matrix and an image vector matrix, for example:

[0054] The target Chinese text is vectorized to obtain a text vector matrix: each word in the target Chinese text is word-embedded: each word in the target Chinese text is vectorized to obtain a first vector, where the first vector is used to indicate the semantic information of each word and how it is used in the context; each word in the target Chinese text is position-embedded: the position of each word in the target Chinese text is vectorized to obtain a second vector, where the second vector is used to indicate the position order and position weight of each word in the target Chinese text; the target Chinese text is paragraph-embedded: each sentence in the target Chinese text is vectorized to obtain a third vector, where the third vector is used to assign a unique paragraph identifier to each sentence; based on the first vector, the second vector, and the third vector, a text vector matrix is ​​obtained. This method can capture the position and paragraph structure information of words in the text, which is particularly important for sentiment analysis of long texts, because it can distinguish the sentiment tendencies of different sentences and avoid the simplification of global sentiment.

[0055] When performing paragraph embedding processing on the target Chinese text, the sentences in the target Chinese text are determined in the following way: the target Chinese text is divided according to the attribute words, and the text on the left of the attribute word is used as the first sentence, and the text on the right of the attribute word is used as the second sentence. This sentence division method based on attribute words can more accurately locate the core part of emotional expression, and is particularly effective for emotional analysis of texts such as comments and feedback, because it can directly focus on the user's emotional attitude towards a specific attribute.

[0056] The dependency glyph graph is vectorized to obtain an image vector matrix: the dependency glyph graph is divided into multiple image regions containing glyph images of a single Chinese character, wherein each image region corresponds to the glyph information of a Chinese character; each image region is encoded to obtain a vector representation of each image region; the vector representations of each image region are spliced ​​to obtain an initial image vector matrix; the initial image vector matrix is ​​processed based on a preset linear layer to obtain an image vector matrix, wherein the linear layer is used to adjust the dimension and feature representation of the initial image vector matrix. This method can effectively process Chinese character information in an image, and is particularly useful for sentiment analysis of text with handwriting or artistic fonts, because it can capture the influence of font style on emotional expression.

[0057] The following is a specific embodiment:

[0058] Use the multimodal language model to vectorize the target Chinese text and the dependency glyph graph respectively to obtain the text vector matrix and image vector matrix:

[0059] Use a multimodal language model to vectorize the target Chinese text (for example, through the Bidirectional Encoder Representations from Transformers (BERT) model in the multimodal language model) to obtain a text vector matrix: Get the target Chinese text. First, divide the target Chinese text into the part on the left side of the attribute word and the part on the right side of the attribute word. The standard form input into BERT is:

[0060] [CLS]+the part on the left of the attribute word (i.e. the first sentence above)+the attribute word+[SEP]+the attribute word+the part on the right of the attribute word (i.e. the second sentence above)+[SEP]. [CLS] is a special classification tag in BERT, and [SEP] is a separation tag in BERT. The above standard forms are processed by token embedding, position embedding, and segment embedding respectively. In the BERT model, the final input representation is the total input representation obtained by adding the word embedding, position embedding, and paragraph embedding. Such an input representation enables the model to simultaneously consider the semantics of the word, the position of the word, and the paragraph information to which the word belongs. Through the format of [CLS]+the part on the left of the attribute word+the attribute word+[SEP]+the attribute word+the part on the right of the attribute word+[SEP], BERT can not only understand the meaning of each word, but also understand the structure between words and the context of the attribute word based on the position and paragraph ID information, thereby improving the accuracy of attribute-level sentiment classification. The total input representation is sent to the Encoder (a neural network component used to convert input data into hidden layer representation (or feature representation)). The backbone of the Encoder is a multi-layer Transformer network. The Transformer model uses a self-attention mechanism to process the input sequence, allowing the model to work in parallel when processing sequence data, thereby significantly improving the training efficiency. First, the total input vector will be converted into H0 = [x1, x2...xx], and then put into the 24-layer Transformer network. The encoding output of each layer of Transformer is as follows, and the text vector matrix is ​​finally obtained:

[0061]

[0062] in, represents the encoded output of layer I, Represents the encoded output of layer I-1, and TransformerBlock represents a layer of Transformer network.

[0063] For the lth Transformer layer, the self-attention head Al is calculated as follows:

[0064] Q=H l-1 W l Q ,K=H l-1 W l K ,V=H l-1 W l V ,

[0065]

[0066] in, represents the query vector weight matrix of the l-1th layer, represents the weight matrix of the vector being checked at the l-1th layer, represents the value vector weight matrix of the l-1th layer, Q represents the query vector of the l-1th layer, K represents the query vector of the l-1th layer, V represents the query vector of the l-1th layer, softmax represents the normalized exponential function, d k is the dimension of the hidden layer.

[0067] Text vector matrix

[0068] Among them, token embedding is the process of converting each word (each token, a token can be a character, word, subword or special mark) in the target Chinese text into a high-dimensional vector. This vector representation not only contains the semantic information of the word, but also takes into account how it is used in context. In BERT, word embedding is achieved through pre-trained word vectors, which are learned on large-scale text datasets and can capture the rich semantic information and contextual dependencies of the word. Position embedding is another type of vector representation that provides specific position information of the word in the sentence. Since the Transformer model does not rely on recursive processing of sequences, but processes the input sequence through a self-attention mechanism, the model itself cannot capture the order information of the words. Position embedding solves this problem. It gives each position in the sequence a unique vector representation, allowing the model to distinguish the order of words and understand the structure of the sentence. Position embedding is usually added to the word embedding at the model input stage so that the model can consider both the semantics and position of the word when processing. Paragraph embeddings are vector representations used to distinguish different paragraphs in an input sequence, which is particularly important when dealing with inputs of two or more sentences (such as question-answer pairs, sentence pair classification, and other tasks). In BERT, when the input contains two sentences (for example, the text on the left and right sides of an attribute word split in a sentiment analysis task), a specific paragraph ID is assigned to each sentence, and the corresponding paragraph embedding is added to the word embedding. This enables the model to identify sentence boundaries and understand the relationship between two sentences, which is critical for tasks such as associating sentence pairs and sentiment analysis.

[0069] The dependency glyph graph is vectorized using the multimodal language model to obtain an image vector matrix (for example, the dependency glyph graph is vectorized using the Vision Transformer (ViT) in the multimodal language model as an image editor to obtain an image vector matrix):

[0070] A preset linear layer is added to replace the image classification layer to match the dimension of the features obtained by the text encoder. Given a dependency glyph graph of a target Chinese text, it is input into ViT, and the dependency glyph graph is segmented into multiple image regions containing glyph images of a single Chinese character, where each image region corresponds to the glyph information of a Chinese character; each image region is encoded to obtain a vector representation of each image region; the vector representations of each image region are concatenated to obtain an initial image vector matrix; the initial image vector matrix is ​​processed based on the preset linear layer to obtain an image vector matrix, thereby obtaining an image representation (i.e., the above-mentioned image vector matrix) H V , for example, is determined by the following formula:

[0071] H V=ViT(I)+W I ,

[0072] Among them, H V represents the image vector matrix, I represents the dependency graph of the target Chinese text, ViT is the image editor, and W I Represents the additional linear layer weights.

[0073] Step S208, determining the attention scores of the text vector matrix and the image vector matrix respectively.

[0074] In the technical solution provided in step S208, there are multiple ways to determine the attention scores of the text vector matrix and the image vector matrix respectively, for example: determine the image vector corresponding to the text vector in the text vector matrix in the image vector matrix to obtain the text image pair; calculate the similarity between the text vector and the image vector in each text image pair; assign attention weights to the text vector in the text vector matrix and the corresponding image vector in the image vector matrix based on the similarity; determine the text attention score and the image attention score based on the attention weights. The introduction of the attention mechanism enables the model to automatically learn the importance of text and image information, which is particularly effective for processing complex text containing multimodal information because it can highlight the parts that contribute most to sentiment classification.

[0075] The following is a specific embodiment:

[0076] Get the text vector matrix H T and the image vector matrix H V Finally, a multimodal interaction model is used to learn the multimodal hidden representation (i.e., the above-mentioned multimodal representation feature) H. The model is built based on the Self-Attention mechanism (Self-Attention mechanism is a method for dynamically assigning weights between elements within the model in sequence data processing, especially in the fields of natural language processing and image processing.) It associates different positions of the target Chinese text with different areas of the dependency graph by calculating the attention score of each text and image pair separately. Therefore, internal structure and external relationship information can be obtained from text and visual modalities. The modal hidden representation is obtained jointly by image-aware text attention learning and text-aware image attention learning.

[0077] For example, the text attention score can be obtained by the following formula:

[0078] H I→T =att(Q=H T ,K=H I ,V=H I ),

[0079] Among them, att is a weight matrix, which is used to project the text representation into the vector space associated with the image features and convert the output into attention weights, whose value range is in (0,1), indicating the importance of each text feature.

[0080] The image attention score can be obtained by the following formula:

[0081] H T→I =att(Q=H I ,K=H T ,V=H T ),

[0082] Among them, att is a weight matrix, which is used to project the text representation into the vector space associated with the image features and convert the output into attention weights, whose value range is in (0,1), which represents the importance of each image feature.

[0083] Step S210, determining the sentiment polarity of the target Chinese text based on the attention score, the text vector in the text vector matrix, and the image vector in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

[0084] In the technical solution provided in step S210, there are many ways to determine the sentiment polarity of the target Chinese text by the text vectors in the text vector matrix and the image vectors in the image vector matrix, for example: weighting the text vector matrix and the image vector matrix according to the attention score to obtain a multimodal representation feature; inputting the multimodal representation feature into the decision layer to obtain a sentiment polarity score, wherein the decision layer is used to convert the multimodal representation feature into a score or probability representing the sentiment polarity; when the sentiment polarity score is greater than or equal to a preset score threshold, the sentiment of the target Chinese text is determined to be positive, and when the sentiment polarity score is less than the preset score threshold, the sentiment of the target Chinese text is determined to be negative. In practical applications, this method can effectively improve the accuracy of text sentiment analysis, especially when processing Chinese texts with complex contexts and polysemous words, and it is particularly outstanding, providing strong technical support for social media analysis, market sentiment monitoring, customer service and other fields, and also opening up new directions for the research of text understanding and natural language processing.

[0085] The following is a specific embodiment: the text vector matrix and the image vector matrix are weighted according to the attention score to obtain a multimodal representation feature. For example, through the following formula:

[0086]

[0087] Among them, H represents the multimodal representation feature, H I→T represents the image attention score, H T→I Represents the text attention score.

[0088] Use the multi-layer perceptron in the multimodal interaction model to predict polarity. For example, first determine the sentiment polarity vector H using the following formula: P :

[0089]

[0090] in, Represents the use of a multi-layer perceptron in a multimodal interaction model to predict the polarity vector H P .

[0091] Get the sentiment polarity vector H P After that, it is input into a softmax layer (decision layer) and the sentiment polarity score is obtained by the following formula:

[0092] P P =softmax(W p H P +B P ),

[0093] in, W p and B P are all preset model parameters, P P represents the sentiment polarity score, P P If it is greater than or equal to 0.5, the sentiment is judged to be positive, otherwise it is judged to be negative.

[0094] The following is an evaluation of the multimodal language model in the embodiment of the present application on a Chinese data set. Table 1 shows the basic information statistics of the data set. There are three fields in the data set: groceries (Chan), furniture (Furn), and kitchen (Kith). 3600 comments are selected from each field, 60% of the comments are used as training data, 20% of the comments are used as verification data, and the remaining comments are used as test data, as shown in Table 1. The distribution of polarity (i.e., positive and negative distribution) is balanced in all fields.

[0095] Table 1

[0096]

[0097] After multiple adjustments, the relevant parameters that achieve the best effect on the validation set are shown in Table 2.

[0098] Table 2

[0099]

[0100] The F1 value is mainly used to evaluate the classification results. The relevant formula is as follows:

[0101] P=TP / (FP+TP), R=TP / (TP+FN), F1=2*P*R / (P+R);

[0102] Among them, F1 represents the result of the evaluation classification, TP is the number of correctly predicted positive examples, FN is the number of positive examples predicted as negative examples, FP is the number of negative examples predicted as positive examples, P is the precision, and R is the recall.

[0103] The existing prediction models are:

[0104] ATAE-LSTM: The attribute word vectors are concatenated in both the input layer and the hidden layer of the LSTM output and the attention mechanism is used to obtain the final result.

[0105] Memory Network (MemNet): MemNet is a long-term memory model that combines a neural attention model with external memory to calculate the importance of each context word to an attribute word.

[0106] TD-BERT: Use BERT to model the preceding and following context of attribute words for attribute-level sentiment classification. The input to BERT is in the form of [CLS] + the left part of the attribute word + the attribute word + [SEP] + the attribute word + the right part of the attribute word + [SEP].

[0107] Relation-aware Graph Attention Networks (R-GAT): A relational graph attention model is used to encode the dependency tree structure of attribute words for attribute-level sentiment classification.

[0108] Radical And Font-based Granularity for Chinese Text Classification (RAFG): uses multiple granular features (e.g., radicals, characters, and words) to represent text classification for Chinese text. It is used for attribute-level sentiment classification.

[0109] Character Logogram Model (CLM): A bidirectional CNN model is used to learn information from the symbol pixel matrix converted from Chinese text to perform attribute-level sentiment classification.

[0110] Chinese-specific BERT model with phonetic and glyph information (ChineseBERT): A model that adds phonetic and glyph information of Chinese characters to BERT pre-training. The phonetic vector of a Chinese character is obtained from the phonetic character sequence, and then several different fonts form the glyph vector. Finally, the phonetic vector, glyph vector and word vector are fused together to obtain the final vector and input into the pre-trained model.

[0111] Dual Graph Convolutional Network (DUAL-GCN): It considers the complementarity of semantic relations and grammatical structures at the same time. It proposes a Syntactic Graph Convolutional Network module (SynGCN module) with rich syntactic information to balance the errors of dependency parsing, and designs a SemGCN module with self-attention to obtain semantic relations.

[0112] Attribute-centered learning method (ACLT) for Chinese attribute-level sentiment classification based on tree-structured self-attention: First, the sentence and attribute words are input into BERT to obtain text encoding representation, and then the probability of attribute words as root nodes and the weights of the edges of the syntax tree are calculated. Two loss functions are defined, one to maximize the probability of attribute words becoming root nodes and the other to minimize the error of attribute word sentiment classification. In the learning process, the dependency tree is made to spontaneously associate attribute words with sentiment words.

[0113] By comparing the results obtained by the multimodal language model of the present application with the above-mentioned models, it can be concluded as shown in Table 3, where the benchmark models can be divided into three categories, the first category is traditional text-based methods, such as ATAE-LSTM, TD-BERT and R-GAT. The second category is methods that integrate glyph or pinyin information for sentiment classification, such as TAFG, CLM, ChineseBERT. The third category is methods that use dependency relationships, such as R-GAT, DUAL-GCN and ACLT. Compared with other traditional text-based methods, RGAT has the best effect. This shows that dependency trees are conducive to capturing the relationship between attribute words and sentiment words. In addition, the effects of CLM and RAFG are lower than those of text-based models, which shows that using only radical information or images cannot perform attribute-level sentiment classification tasks well. In addition, ChineseBERT outperforms other benchmark models, which shows that it is necessary to use text and visual modalities to learn glyph information at the same time. The model (OURS) in this application is significantly better than ChineseBERT and other benchmark models, which proves the effectiveness of dependency glyph graphs in Chinese text attribute-level sentiment classification tasks.

[0114] Table 3

[0115]

[0116] The influence of different factors on the model of this application is shown in Table 4. Text means that only BERT is used to encode the text information to obtain the final result, and Vision means that only ViT is used to encode the dependent image information of the comment text. From the results, it can be found that the effect of the text modality is better than that of the visual modality. This also confirms that the visual modality cannot be used alone to encode the dependent glyph graph for attribute-level sentiment classification, and it also shows that the role of text information in attribute-level sentiment classification tasks is still the greatest. In addition, the performance of text modality + visual modality is better than that of a single text modality or visual (image) modality, which shows that integrating text and visual (image) modalities for attribute-level sentiment classification is effective. In addition, the model is better than other models, proving the effectiveness of the multimodal interaction model, which can better capture information from text and visual modalities.

[0117] Table 4

[0118]

[0119] Analyze the impact of dependencies on the results, because dependencies are very important for the external structure between attribute words and context in dependency character images. As shown in Table 5, R-GAT, DUAL-GCN and ACLT are three different graph neural models that use dependency trees, which have achieved very advanced performance in attribute-level sentiment classification tasks. In the model of the embodiment of the present application, Dep means converting the entire dependency tree into a dependency glyph graph and combining it with the text representation, while AspectDep means converting only the dependency tree for attribute words into a dependency glyph graph and then combining it with the text representation.

[0120] Table 5

[0121]

[0122] The results show that the glyph image multimodal model with dependency relationships achieves better performance than the graph neural network model. This shows that images are a natural and simple way to capture dependency relationships, and glyph images with dependency relationships can capture the external structure between attribute words and context. In addition, using a dependency tree for attribute words in a dependency image is more effective than using the entire dependency tree, which proves that when performing attribute-level sentiment classification tasks, although it is helpful to consider too much about the relationship between all words, it is better to only consider the relationship between attribute words and other words that are closely related to the attribute words.

[0123] The following are some of the judgment results of the model in the embodiment of the present application and the TD-BERT model, and they are compared and analyzed. The results are shown in Table 6, and the underlined ones are the attribute words and the corresponding sentiment words in this sentence. In the first sentence, the attribute word "sanitation condition" corresponds to the sentiment word "worry", and the font information in "worry" successfully helps the model make a correct judgment on the sentiment. At the same time, the attribute words and sentiment words of these three sentences have a relatively short distance. It can be seen that the model can better judge the sentiment corresponding to the attribute words due to the existence of the dependency graph.

[0124] Table 6

[0125]

[0126] Through the above steps, it is possible to achieve a more comprehensive understanding of emotional expression and improve classification accuracy by fusing text and image information. This method can process text with rich visual elements, such as social media comments with emoticons, and can improve classification accuracy. The multimodal Chinese sentiment classification method combines the semantic information of the text and the glyph image information of the Chinese characters to construct a more comprehensive and detailed text representation, which can more accurately capture the emotional polarity of Chinese text.

[0127] This method not only takes into account the grammatical structure and context of the text, but also integrates the visual features of Chinese characters, greatly improving the accuracy of sentiment classification, especially when dealing with Chinese texts with complex contexts and polysemous words. In addition, by introducing the attention mechanism, it can automatically assign weights to text and image information, further enhancing the adaptability and flexibility of the model, making sentiment classification more accurate and efficient. In practical applications, this method can effectively improve the accuracy of text sentiment analysis and provide strong technical support for social media analysis, market sentiment monitoring, customer service and other fields. In fields such as education, art, and advertising that require a deep understanding of text emotions and visual expressions, this method can also play an important role, helping people better understand and analyze the emotional information contained in the text, and promoting the development and application of emotional intelligence technology.

[0128] The present application also provides a schematic diagram of the structure of a multimodal language model. Figure 3As shown, the structure of the multimodal language model in the embodiment of the present application is shown. The model mainly includes three parts: a text encoder (using the BERT model), an image encoder (using ViT), and a multimodal interaction model. The comment text (target Chinese text, for example, the dessert tastes delicious) is segmented and fed into BERT for encoding, and the dependency structure is converted into a dependency image and encoded by ViT. Subsequently, the encoded text and image information are fused in the multimodal interaction model (MultimodalInteraction), and sentiment classification prediction is performed through a multilayer perceptron to obtain sentiment polarity (Pos represents positive sentiment, and Neg represents negative sentiment).

[0129] The present application also provides a schematic diagram of the structure of a multimodal interaction model. Figure 5 As shown, the structure of the multimodal interaction model in the embodiment of the present application is shown, and how to perform information interaction and fusion between text modality and visual (image) modality (based on the text vector in the text vector matrix and the image vector corresponding to the image vector in the image vector matrix, the text image pair is obtained). The multimodal interaction model calculates the text attention score (Text-aware Image Attention) and the image attention score (Image-aware Image Attention) respectively through the attention mechanism, thereby associating different positions of the comment text with different areas of the dependent image. Specifically, the image-aware text attention learning and the text-aware image attention learning work together to produce a fused multimodal hidden representation H (i.e., the above-mentioned multimodal representation feature). This model design allows the model to simultaneously obtain internal structure and external relationship information from text and visual modalities, thereby improving the performance of attribute-level sentiment classification tasks. The Add&Norm in the figure is the residual connection and layer normalization module in the Transformer, and the Feed Forward is the feedforward layer module in the Transformer.

[0130] The present application also provides a schematic diagram of the structure of a multi-modal Chinese sentiment classification device. Figure 6 As shown, including:

[0131] The acquisition module 602 is used to acquire the target Chinese text.

[0132] The first determination module 604 is used to determine the attribute words of the target Chinese text and construct a dependency glyph graph of the attribute words based on the attribute words, wherein the dependency glyph graph includes the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text.

[0133] The first determination module 604 is also used to segment the target Chinese text and mark the part of speech; based on the part of speech, identify the words in the target Chinese text that belong to the preset part of speech as candidate attribute words, where the preset part of speech is noun; screen the candidate attribute words based on the context to determine the attribute words of the target Chinese text.

[0134] The first determination module 604 is also used to parse the grammatical structure of the target Chinese text to obtain the grammatical dependency relationship of each word in the target Chinese text; determine the dependency tree based on the attribute word and the grammatical dependency relationship: use the attribute word as the root node of the dependency tree, identify the words directly connected to the attribute word based on the grammatical dependency relationship, and use them as first-level child nodes, which are used to reflect direct grammatical associations; traverse the dependency relationship, add all words that have grammatical dependency relationships with the attribute word as child nodes, and obtain a dependency tree; embed the glyph image information of each Chinese character into the dependency tree to obtain a dependency glyph graph, wherein the glyph image information is used to display the visual shape expression of Chinese characters in one or more fonts.

[0135] The vectorization module 606 is used to perform vectorization processing on the target Chinese text and the dependent glyph graph respectively to obtain a text vector matrix and an image vector matrix.

[0136] The vectorization module 606 is also used to vectorize the target Chinese text to obtain a text vector matrix: perform word embedding processing on each word in the target Chinese text: vectorize each word in the target Chinese text to obtain a first vector, wherein the first vector is used to indicate the semantic information of each word and the method of use in the context; perform position embedding processing on each word in the target Chinese text: vectorize the position of each word in the target Chinese text to obtain a second vector, wherein the second vector is used to indicate the position order and position weight of each word in the target Chinese text; perform paragraph embedding processing on the target Chinese text: vectorize each sentence in the target Chinese text to obtain a third vector, wherein the third vector is used to assign a unique paragraph identifier to each sentence; based on the first vector, the second vector, and the third vector, a text vector matrix is ​​obtained. In some embodiments of the present application, when performing paragraph embedding processing on the target Chinese text, the sentences in the target Chinese text are determined in the following manner: the target Chinese text is divided according to the attribute words, the text on the left side of the attribute words is used as the first sentence, and the text on the right side of the attribute words is used as the second sentence.

[0137] The vectorization module 606 is also used to perform vectorization processing on the dependency glyph graph to obtain an image vector matrix: divide the dependency glyph graph into multiple image areas containing glyph images of a single Chinese character, wherein each image area corresponds to the glyph information of a Chinese character; encode each image area to obtain a vector representation of each image area; splice the vector representations of each image area to obtain an initial image vector matrix; process the initial image vector matrix based on a preset linear layer to obtain an image vector matrix, wherein the linear layer is used to adjust the dimension and feature representation of the initial image vector matrix.

[0138] The second determination module 608 is used to determine the attention scores of the text vector matrix and the image vector matrix respectively.

[0139] The second determination module 608 is also used to determine the image vector corresponding to the text vector in the text vector matrix in the image vector matrix to obtain a text-image pair; calculate the similarity between the text vector and the image vector in each text-image pair; assign attention weights to the text vector in the text vector matrix and the corresponding image vector in the image vector matrix based on the similarity; and determine the text attention score and the image attention score based on the attention weights.

[0140] The third determination module 610 is used to determine the sentiment polarity of the target Chinese text based on the attention score, the text vector in the text vector matrix, and the image vector in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

[0141] The third determination module 610 is also used to weight the text vector matrix and the image vector matrix according to the attention score to obtain multimodal representation features; input the multimodal representation features into the decision layer to obtain a sentiment polarity score, wherein the decision layer is used to convert the multimodal representation features into a score or probability representing sentiment polarity; when the sentiment polarity score is greater than or equal to a preset score threshold, the sentiment of the target Chinese text is determined to be positive, and when the sentiment polarity score is less than the preset score threshold, the sentiment of the target Chinese text is determined to be negative.

[0142] It should be noted that the various modules in the above-mentioned multimodal Chinese sentiment classification device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0143] The embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above multimodal Chinese sentiment classification method. For example, obtain the target Chinese text; determine the attribute words of the target Chinese text, and construct a dependency glyph graph of the attribute words based on the attribute words, wherein the dependency glyph graph contains the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; vectorize the target Chinese text and the dependency glyph graph respectively to obtain a text vector matrix and an image vector matrix; determine the attention scores of the text vector matrix and the image vector matrix respectively; determine the sentiment polarity of the target Chinese text based on the attention score, the text vectors in the text vector matrix, and the image vectors in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

[0144] The embodiment of the present application also provides an electronic device, which includes a processor, and the processor is used to run a program, wherein the above multimodal Chinese sentiment classification method is executed when the program is running. For example, a target Chinese text is obtained; the attribute words of the target Chinese text are determined, and a dependency glyph graph of the attribute words is constructed based on the attribute words, wherein the dependency glyph graph contains the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; the target Chinese text and the dependency glyph graph are vectorized respectively to obtain a text vector matrix and an image vector matrix; the attention scores of the text vector matrix and the image vector matrix are determined respectively; the sentiment polarity of the target Chinese text is determined based on the attention score, the text vectors in the text vector matrix, and the image vectors in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

[0145] According to another aspect of the embodiment of the present application, a computer program product is also provided, including a computer program, which implements the above multimodal-based Chinese sentiment classification method when executed by a processor. For example, a target Chinese text is obtained; the attribute words of the target Chinese text are determined, and a dependency glyph graph of the attribute words is constructed based on the attribute words, wherein the dependency glyph graph contains the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; the target Chinese text and the dependency glyph graph are vectorized respectively to obtain a text vector matrix and an image vector matrix; the attention scores of the text vector matrix and the image vector matrix are determined respectively; the sentiment polarity of the target Chinese text is determined based on the attention score, the text vectors in the text vector matrix, and the image vectors in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

[0146] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0151] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A Chinese sentiment classification method based on multimodality, characterized in that: include: Get the target Chinese text; Determine the attribute words of the target Chinese text, and construct a dependency graph of the attribute words based on the attribute words, wherein the dependency graph contains the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; Performing vectorization processing on the target Chinese text and the dependency glyph graph respectively to obtain a text vector matrix and an image vector matrix; Determining attention scores of the text vector matrix and the image vector matrix respectively; The sentiment polarity of the target Chinese text is determined based on the attention score, the text vector in the text vector matrix, and the image vector in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

2. The method according to claim 1, characterized in that Determining the attribute words of the target Chinese text includes: Segmenting the target Chinese text and marking the parts of speech; Based on the part of speech, identifying words in the target Chinese text that belong to a preset part of speech as candidate attribute words, wherein the preset part of speech is a noun; The candidate attribute words are screened based on the context to determine the attribute words of the target Chinese text.

3. The method according to claim 1, characterized in that Constructing a dependency graph of the attribute word based on the attribute word includes: Parsing the grammatical structure of the target Chinese text to obtain the grammatical dependency relationship of each word in the target Chinese text; Determine the dependency tree based on the attribute word and the grammatical dependency relationship: take the attribute word as the root node of the dependency tree, identify the words directly connected to the attribute word based on the grammatical dependency relationship as first-level child nodes, and the first-level child nodes are used to reflect direct grammatical associations; traverse the dependency relationship, add all words that have grammatical dependencies with the attribute word as child nodes, and obtain the dependency tree; The glyph image information of each Chinese character is embedded into the dependency tree to obtain the dependency glyph graph, wherein the glyph image information is used to display the visual shape expression of Chinese characters in one or more fonts.

4. The method according to claim 1, characterized in that: The target Chinese text and the dependency glyph graph are respectively vectorized to obtain a text vector matrix and an image vector matrix, including: The target Chinese text is vectorized to obtain a text vector matrix: Performing word embedding processing on each word in the target Chinese text: vectorizing each word in the target Chinese text to obtain a first vector, wherein the first vector is used to indicate the semantic information of each word and the usage method in the context; Performing position embedding processing on each word in the target Chinese text: vectorizing the position of each word in the target Chinese text to obtain a second vector, wherein the second vector is used to indicate the position order and position weight of each word in the target Chinese text; Performing paragraph embedding processing on the target Chinese text: vectorizing each sentence in the target Chinese text to obtain a third vector, wherein the third vector is used to assign a unique paragraph identifier to each sentence; The text vector matrix is ​​obtained based on the first vector, the second vector and the third vector.

5. The method according to claim 4, characterized in that When performing paragraph embedding processing on the target Chinese text, the sentences in the target Chinese text are determined in the following manner: The target Chinese text is divided according to the attribute words, the text on the left side of the attribute words is taken as the first sentence, and the text on the right side of the attribute words is taken as the second sentence.

6. The method according to claim 1, characterized in that The target Chinese text and the dependency glyph graph are respectively vectorized to obtain a text vector matrix and an image vector matrix, including: The dependency glyph graph is vectorized to obtain an image vector matrix: Segmenting the dependency glyph graph into a plurality of image regions containing glyph images of a single Chinese character, wherein each of the image regions corresponds to glyph information of a Chinese character; Encoding each of the image regions to obtain a vector representation of each of the image regions; splicing the vector representations of each of the image regions to obtain an initial image vector matrix; The initial image vector matrix is ​​processed based on a preset linear layer to obtain the image vector matrix, wherein the linear layer is used to adjust the dimension and feature representation of the initial image vector matrix.

7. The method according to claim 1, characterized in that Determine the attention scores between the text vector matrix and the image vector matrix respectively: Determine the image vector corresponding to the text vector in the text vector matrix in the image vector matrix to obtain a text-image pair; Calculating the similarity between the text vector and the image vector in each of the text-image pairs; assigning attention weights to text vectors in the text vector matrix and corresponding image vectors in the image vector matrix based on the similarity; A text attention score and an image attention score are determined according to the attention weights.

8. The method according to claim 7, characterized in that Determining the sentiment polarity of the target Chinese text based on the attention score, the text vector in the text vector matrix, and the image vector in the image vector matrix includes: Weighting the text vector matrix and the image vector matrix according to the attention score to obtain a multimodal representation feature; Inputting the multimodal representation features into a decision layer to obtain a sentiment polarity score, wherein the decision layer is used to convert the multimodal representation features into a score or probability representing sentiment polarity; When the sentiment polarity score is greater than or equal to a preset score threshold, the sentiment of the target Chinese text is determined to be positive; when the sentiment polarity score is less than the preset score threshold, the sentiment of the target Chinese text is determined to be negative.

9. A Chinese sentiment classification device based on multimodality, characterized in that: include: An acquisition module is used to acquire the target Chinese text; A first determination module is used to determine the attribute words of the target Chinese text, and construct a dependency graph of the attribute words based on the attribute words, wherein the dependency graph includes the glyph image information of each Chinese character in the target Chinese text and a dependency tree reflecting the grammatical dependency relationship between the attribute words and other words in the target Chinese text; A vectorization module, used for performing vectorization processing on the target Chinese text and the dependency glyph graph respectively to obtain a text vector matrix and an image vector matrix; A second determination module, used to determine the attention scores of the text vector matrix and the image vector matrix respectively; The third determination module is used to determine the sentiment polarity of the target Chinese text based on the attention score, the text vector in the text vector matrix and the image vector in the image vector matrix, wherein the sentiment polarity includes positive sentiment or negative sentiment.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the multimodal Chinese sentiment classification method described in any one of claims 1 to 8.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program, when running, executes the multimodal Chinese sentiment classification method described in any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by the processor, the multimodal Chinese sentiment classification method described in any one of claims 1 to 8 is implemented.

Images