Language sentiment prediction method and device, storage medium and electronic equipment

Abstract

The present disclosure provides a language sentiment prediction method and device, a storage medium and an electronic device, and relates to the technical field of natural language processing. The method comprises: obtaining a to-be-processed sequence, obtaining a syntax dependency type of a word in the to-be-processed sequence and an attribute perception distance of the word; performing embedding processing to obtain context representation information of the word, syntax dependency representation information and attribute perception distance representation information, and initial representation information of an attribute word; processing by using an attention mechanism, generating an adjacency matrix according to a processing result; obtaining aggregated representation information of the word by aggregating the context representation information, the syntax dependency representation information and the attribute perception distance representation information; determining a related word of the attribute word, fusing aggregated representation information of the related word based on the adjacency matrix to obtain final representation information of the attribute word; and processing by using a sentiment prediction model to obtain a sentiment prediction result of the attribute word. The present disclosure is beneficial to improving the accuracy of sentiment prediction.

Description

Technical Field

[0001] This disclosure relates to the field of natural language processing technology, and in particular to a language sentiment prediction method, a language sentiment prediction device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] Sentiment analysis or prediction is an important research area in natural language processing, aiming to identify the specific sentiment classification of words in a sentence, such as whether the sentiment polarity is positive, negative, or neutral.

[0003] In some related technologies, traditional machine learning models are used to predict the sentiment of words in a sentence. This relies on the quality of human-designed features. When the quality of features is not high, it seriously affects the accuracy of sentiment prediction. Moreover, human-designed features are very costly in terms of manpower and time. Summary of the Invention

[0004] This disclosure provides a language sentiment prediction method, a language sentiment prediction device, a computer-readable storage medium, and an electronic device, to at least partially solve the problem of low accuracy in sentiment prediction in related technologies.

[0005] According to a first aspect of this disclosure, a language sentiment prediction method is provided, comprising: acquiring a processing sequence formed by a statement to be processed and attribute words in the statement to be processed; acquiring the syntactic dependency type of the words in the processing sequence and the attribute-aware distance of the words, wherein the attribute-aware distance of the words represents the syntactic distance between the words and the attribute words; performing embedding processing on the processing sequence, the syntactic dependency type of the words, and the attribute-aware distance of the words to obtain contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the words; obtaining initial representation information of the attribute words based on the contextual representation information of the words; and utilizing annotation... The intention mechanism processes the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute word, and generates an adjacency matrix based on the processing results. It then aggregates the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word to obtain aggregated representation information of the word. Next, it identifies the associated words of the attribute word within the word, and fuses the aggregated representation information of the associated words based on the adjacency matrix to obtain the final representation information of the attribute word. Finally, it processes the final representation information of the attribute word using a sentiment prediction model to obtain the sentiment prediction result of the attribute word.

[0006] According to a second aspect of this disclosure, a language sentiment prediction device is provided, comprising: an information acquisition module configured to acquire a processing sequence formed by a statement to be processed and attribute words in the statement to be processed, acquire syntactic dependency types of words in the processing sequence and attribute-aware distances of the words, wherein the attribute-aware distances of the words represent the syntactic distances between the words and the attribute words; an embedding processing module configured to perform embedding processing on the processing sequence, the syntactic dependency types of the words, and the attribute-aware distances of the words to obtain contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the words; and obtain initial representation information of the attribute words based on the contextual representation information of the words; and an adjacency relationship processing module configured to... The system is configured to use an attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words, and generate an adjacency matrix based on the processing results. A multi-information aggregation module is configured to aggregate the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the words to obtain aggregated representation information of the words; determine the associated words of the attribute words in the words, and fuse the aggregated representation information of the associated words based on the adjacency matrix to obtain the final representation information of the attribute words; a prediction processing module is configured to use a sentiment prediction model to process the final representation information of the attribute words to obtain the sentiment prediction result of the attribute words.

[0007] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the language sentiment prediction method of the first aspect and its possible implementations.

[0008] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the language sentiment prediction method of the first aspect and possible implementations thereof by executing the executable instructions.

[0009] The technical solution disclosed herein has the following beneficial effects:

[0010] On the one hand, by combining information from various aspects such as the sentence to be processed, attribute words in the sentence, syntactic dependency types of words, and attribute perception distance of words, this approach fully mines and utilizes the information within the sentence. Furthermore, by introducing attention mechanisms and mechanisms for aggregation and fusion of representational information, it can effectively integrate multi-faceted information, fully learn the attribute-related information and global syntactic and semantic information in the sentence, and generate a final representation with a large amount of information, thus providing comprehensive information for sentiment prediction and improving its accuracy. On the other hand, the information required by this approach consists entirely of objective information such as words and syntax in the sentence, eliminating the need for feature design. The accuracy of sentiment prediction does not depend on the quality of features, thereby saving manpower and time costs and ensuring the robustness of sentiment prediction. Moreover, this approach can be applied to various scenarios, especially performing well in scenarios where dictionary or syntactic resources are scarce. Attached Figure Description

[0011] Figure 1 A flowchart of a language sentiment prediction method in this exemplary embodiment is shown.

[0012] Figure 2 A flowchart illustrating one method for generating an adjacency matrix in this exemplary embodiment is shown.

[0013] Figure 3 A flowchart illustrating one method of obtaining a first association relationship is shown in this exemplary embodiment.

[0014] Figure 4 A flowchart illustrating one method of obtaining a second association relationship in this exemplary embodiment is shown.

[0015] Figure 5 A flowchart illustrating one method for obtaining final characterization information in this exemplary embodiment is shown.

[0016] Figure 6 A schematic diagram of the globalization model in this exemplary embodiment is shown.

[0017] Figure 7 This diagram illustrates the structure of a language sentiment prediction device according to this exemplary embodiment.

[0018] Figure 8 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown. Detailed Implementation

[0019] Exemplary embodiments of this disclosure will be described more fully below with reference to the accompanying drawings.

[0020] The accompanying drawings are schematic illustrations of this disclosure and are not necessarily drawn to scale. Some block diagrams shown in the drawings may be functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in hardware modules or integrated circuits, or in networks, processors, or microcontrollers. Implementations can be carried out in various forms and should not be construed as limited to the examples set forth herein. The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough description of embodiments of this disclosure. However, those skilled in the art will recognize that one or more specific details may be omitted when implementing the technical solutions of this disclosure, or other methods, components, apparatuses, steps, etc., may be used to replace one or more specific details.

[0021] Sentiment analysis or prediction is an important research area in natural language processing, aiming to identify the specific sentiment classification of words in a sentence, such as whether the sentiment polarity is positive, negative, or neutral.

[0022] One approach to related technologies involves predicting word sentiment based on language rules or patterns, but this approach is not suitable for scenarios where dictionary or syntactic resources are scarce.

[0023] Another approach to related technologies involves using traditional machine learning models to predict the sentiment of words in a sentence. However, manually designed features significantly impact the performance of the machine learning model and the accuracy of sentiment prediction. When the features are poorly designed, resulting in low feature quality, the accuracy of sentiment prediction is low. Furthermore, manually designing features is extremely costly in terms of manpower and time.

[0024] In another approach to related technologies, deep learning is used, such as employing pre-trained language models, for language sentiment prediction. The inventors have found that current deep learning solutions do not fully utilize sentence information, such as ignoring the attribute-aware distance of words and the syntactic dependency type of words, and that different pieces of information are not effectively integrated. These shortcomings lead to low accuracy in sentiment prediction.

[0025] In view of one or more of the above-mentioned problems, an exemplary embodiment of this disclosure provides a language sentiment prediction method. Figure 1 An exemplary flow of a language sentiment prediction method is shown, which may include the following steps S110 to S140:

[0026] Step S110: Obtain the processing sequence formed by the statement to be processed and the attribute words in the statement to be processed; obtain the syntactic dependency type of the words in the processing sequence and the attribute-aware distance of the words; the attribute-aware distance of the words represents the syntactic distance between the words and the attribute words.

[0027] Step S120: Embedding processing is performed on the sequence to be processed, the syntactic dependency type of the word, and the attribute-aware distance of the word to obtain the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word; the initial representation information of the attribute word is obtained based on the contextual representation information of the word.

[0028] Step S130: Use attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words, and generate an adjacency matrix based on the processing results.

[0029] Step S140: By aggregating the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the words, the aggregated representation information of the words is obtained; among the above words, the associated words of the attribute words are determined, and the aggregated representation information of the associated words is fused based on the adjacency matrix to obtain the final representation information of the attribute words.

[0030] Step S150: The final representation information of the attribute words is processed using the sentiment prediction model to obtain the sentiment prediction results of the attribute words.

[0031] exist Figure 1 The method presented, on the one hand, fully mines and utilizes information from multiple aspects, including the sentence to be processed, attribute words in the sentence, syntactic dependency types of words, and attribute-aware distance of words. Furthermore, by introducing attention mechanisms and representation information aggregation and fusion mechanisms, it can effectively integrate multi-faceted information, fully learn the attribute-related information and global syntactic and semantic information in the sentence, and generate a final representation information with a large amount of information, thus providing comprehensive information for sentiment prediction and improving the accuracy of sentiment prediction. On the other hand, the information required by this scheme is all objective information such as words and syntax in the sentence, without the need for feature design. The accuracy of sentiment prediction does not depend on the quality of features, thereby saving manpower and time costs and ensuring the robustness of sentiment prediction. Moreover, this scheme can be applied to various different scenarios, especially performing well in scenarios where dictionary or syntactic resources are scarce.

[0032] The following is about Figure 1 Each step in the process will be explained in detail.

[0033] refer to Figure 1 In step S110, a processing sequence is obtained, which is formed by the statement to be processed and the attribute words in the statement to be processed. The syntactic dependency type of the words in the processing sequence and the attribute-aware distance of the words are obtained. The attribute-aware distance of the words represents the syntactic distance between the words and the attribute words.

[0034] The statement to be processed can be of any type and obtained through any channel. It can consist of multiple words, including one or more attribute words. An attribute word is the target object of the semantics expressed by the statement; it can be a noun, for example, "appearance" in the statement "This car has a beautiful appearance" is an attribute word. The syntactic dependency type of a word refers to the syntactic relationship between the word and attribute words or other words, such as subject-predicate, verb-object, or adverbial relationships. The attribute-aware distance of a word represents the syntactic distance between the word and its attribute words. This syntactic distance can be the number of words separating the two words in the statement or the distance in the syntactic dependency tree.

[0035] For example, obtain the statement to be processed s = {w1, w2, ..., w m}, w1, w2, etc. represent individual words in the statement, and m represents the length of the statement to be processed, i.e., the total number of words. Determine the attribute words a = {a1, a2, ..., a...} in the statement to be processed. ma}, where a1, a2, etc., represent attribute words in the statement. The statement to be processed (s) and the attribute word (a) are concatenated in the format "[CLS] + statement to be processed + [SEP] + attribute word + [SEP]". The [CLS] flag is usually at the beginning of the sequence, and the [SEP] flag is usually used to separate two sentences. The resulting sequence is:

[0036] I bert ={w cls ,w1,...,w m ,w sep ,a1,...,a ma ,w sep} (1)

[0037] The syntactic dependency type I of each word in the sequence to be processed is obtained using a dependency parsing tool. syn ={syn1,syn2,...,syn n}, where n = m + m a +3. And calculate the attribute-perceived distance I for each word. d ={l1,l2,...,l n The calculation formula is as follows:

[0038]

[0039] Among them, min_dis i C represents the shortest path between an attribute word and the i-th word in a syntactic dependency tree. i =1 indicates that there is a syntactic connection between the attribute word and the i-th word, C i =0 indicates that there is no syntactic connection between them, and L represents the maximum length of the sequence to be processed, which can be n.

[0040] Continue to refer to Figure 1 In step S120, the sequence to be processed, the syntactic dependency type of the word, and the attribute-aware distance of the word are embedded to obtain the context representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word; the initial representation information of the attribute word is obtained based on the context representation information of the word.

[0041] Specifically, the three parts of information—the sequence to be processed, the syntactic dependency type of the word, and the attribute-aware distance of the word—can be embedded separately to obtain the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word.

[0042] For example, the BERT (Bidirectional Encoder Representation from Transformers) model can be used to extract the contextual representation information of words from the sequence to be processed, as shown in the following formula:

[0043]

[0044] in, b b These are learnable model parameters. This represents the output representation of the i-th word from the last layer of BERT. This represents the output representation of the i-th word from an intermediate layer of BERT (e.g., the sixth layer, which is not limited in this disclosure), thereby obtaining the contextual representation information of each word in the sequence to be processed. This includes the contextual representation information of attribute words, which can be used as the initial representation information of attribute words, represented as follows: In one implementation, average pooling or similar methods can be used to fuse the initial representation information of all attribute words to obtain the overall initial representation information of the attribute words. It is also part of the initial representation information of attribute words.

[0045] Similarly, embedding can be performed on the syntactic dependency types of words to obtain the syntactic dependency representation information of each word. The attribute-aware distance of words is embedded to obtain the attribute-aware distance representation information of each word.

[0046] Continue to refer to Figure 1In step S130, the attention mechanism is used to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words, and an adjacency matrix is ​​generated based on the processing results.

[0047] This process utilizes attention mechanisms to further mine information from the contextual representations, syntactic dependency representations, attribute-aware distance representations, and initial representations of attribute words. This allows for the extraction of word-to-word relationships, which can include one or more of the following: relationships between attribute words, relationships between non-attribute words, and relationships between non-attribute words and attribute words. Based on these extracted relationships, an adjacency matrix can be generated. This matrix quantifies the word-to-word relationships and is used for subsequent information fusion between different words.

[0048] In one implementation, reference Figure 2 As shown, the above-mentioned use of attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words, and to generate an adjacency matrix based on the processing results, may include the following steps S210 to S230:

[0049] Step S210: The syntactic awareness attention mechanism is used to process the contextual representation information, syntactic dependency representation information, attribute awareness distance representation information, and initial representation information of attribute words in order to extract the first association relationship between words and attribute words.

[0050] Among them, the syntactic awareness attention mechanism focuses on mining the relationship between words and the whole sentence. Since sentences involve a lot of information, attribute words can be used to approximate the whole sentence. Therefore, the syntactic awareness attention mechanism can extract the first association between words and attribute words, including the association between non-attribute words and attribute words, as well as the association between attribute words.

[0051] In one implementation, reference Figure 3 As shown, the above-mentioned use of syntactic-aware attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words to extract the first association relationship between words and attribute words may include the following steps S310 to S330:

[0052] Step S310: The initial representation information of the attribute word, the syntactic dependency representation information of the word, and the attribute perception distance representation information are aggregated and processed by the gated loop unit to obtain the first intermediate representation information.

[0053] Step S320: After aggregating the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word, the gated loop unit is used for processing to obtain the second intermediate representation information.

[0054] Step S330: Merge the first intermediate representation information and the second intermediate representation information to obtain the first association relationship.

[0055] For example, the extraction process of the first intermediate representation information can refer to the following formula:

[0056]

[0057] The extraction process of the second intermediate representation information can refer to the following formula:

[0058] h m =BiGRU([(h bert W m +λ2h syn );h dis (5)

[0059] The calculation process for the first association relationship can refer to the following formula:

[0060]

[0061] Among them, W a W m b asp These are learnable model parameters. It is the initial representation information of attribute words, which can be obtained by representing the overall attribute words. Repeated n times, h syn It is the syntactic dependency representation information of words, h dis It is the attribute-perceived distance representation information of words, h bert λ1 and λ2 are the contextual representation information of the word, λ1 and λ2 are the hyperparameters of the model, BiGRU(·) represents the processing of the bidirectional gated recurrent unit, and the symbol “;” represents the concatenation operation. a The first intermediate representation information, h, represents the attribute word representation of the syntactic-aware attention output regarding syntactic dependency type and attribute-aware distance. m The second intermediate representation information represents the contextual representation of the syntactic-aware attention output regarding syntactic dependency type and attribute-aware distance. Let be the first association, representing the attention score of the i-th syntactic-aware attention head.

[0062] Step S220: The syntactic awareness self-attention mechanism is used to process the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of words in order to extract the second association relationship between words.

[0063] Among them, the syntactic-aware self-attention mechanism focuses on mining the relationships between individual words (including two identical words). Therefore, the syntactic-aware self-attention mechanism can be used to extract the second relationship between words, including the relationship between non-attribute words and attribute words, the relationship between attribute words and attribute words, and the relationship between non-attribute words and non-attribute words.

[0064] In one implementation, reference Figure 4 As shown, the above-described method of using a syntactic-aware self-attention mechanism to process the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of words to extract the second association relationship between words may include the following steps S410 and S420:

[0065] Step S410: The initial representation information of the attribute words, the syntactic dependency representation information of the words, and the attribute-aware distance representation information are aggregated and then processed by a gated loop unit to obtain query representation information and key-value representation information respectively.

[0066] Step S420: Merge query representation information and key value representation information to obtain the second association relationship.

[0067] In this process, the initial representation information of attribute words, the syntactic dependency representation information of words, and the attribute-aware distance representation information can be aggregated with different parameters, and then processed with different gated loop units to obtain query representation information and key-value representation information.

[0068] For example, query representation information can be calculated using the following formula:

[0069] h q =BiGRU([(h bert W q +λ3h syn );h dis (7)

[0070] Key-value representation information can be calculated using the following formula:

[0071] h k =BiGRU([(h bert W k +λ4h syn );h dis (8)

[0072] The calculation process for the second association relationship can refer to the following formula:

[0073]

[0074] Among them, W q W kh are the learnable parameters of the model. syn It is the syntactic dependency representation information of words, h dis It is the attribute-perceived distance representation information of words, h bert λ3 and λ4 are the contextual representation information of the word, λ3 and λ4 are the hyperparameters of the model, and d is the dimension of the contextual representation information of the word. q It is query representation information, representing the query information (query) output by syntactic-aware self-attention regarding syntactic dependency type and attribute-aware distance, h k It is key-value representation information, representing key-value information of the syntactic-aware self-attention output regarding syntactic dependency type and attribute-aware distance. It is the second association, representing the attention score of the i-th syntactic-aware self-attention head.

[0075] Step S230: Generate an adjacency matrix based on the first association relationship and the second association relationship.

[0076] It should be understood that there may be duplicate association content in the first association and the second association, and different association values ​​may be obtained based on different attention mechanisms. This does not affect subsequent calculations. By merging the first association and the second association, which are results from different attention mechanisms, a more comprehensive association can be obtained to generate an adjacency matrix.

[0077] In one implementation, the first association relationship and the second association relationship can be added together to obtain the adjacency matrix.

[0078] In one implementation, generating an adjacency matrix based on a first association relationship and a second association relationship includes:

[0079] The first and second association relationships are merged to obtain the first intermediate matrix;

[0080] The first intermediate matrix is ​​numerically adjusted based on the attribute-aware distance of words to obtain the second intermediate matrix;

[0081] Generate an adjacency matrix based on the second intermediate matrix.

[0082] For example, the first and second associations can be merged by addition, as shown in the following formula:

[0083]

[0084] in, This represents the attention score of the i-th syntactic-aware attention head. p represents the attention score of the i-th syntactic-aware self-attention head. i Let be the first intermediate matrix, representing the adjacency matrix of the i-th head.

[0085] In sentiment prediction of attribute words, words with closer syntactic distance to the attribute word play a more important role, while words with greater syntactic distance have less influence on the attribute word and may produce noise that misleads the results. The first intermediate matrix can be numerically adjusted based on the attribute-perceived distance of words, as shown in the following formula:

[0086]

[0087] Among them, I d This represents the perceived distance of a word's attributes, where ε is a very small value, such as 10. -9 . This is the second intermediate matrix. By adjusting the first intermediate matrix using formula (11), the weight of words with a greater attribute perception distance in the adjacency matrix can be reduced, or even approached to 0, while the weight of words with a greater attribute perception distance in the adjacency matrix can be increased.

[0088] In one implementation, the second intermediate matrix can be used as an adjacency matrix. Alternatively, the second intermediate matrix can undergo further numerical normalization, as shown in the following formula:

[0089]

[0090] in, This represents the final semantic and syntactic awareness adjacency matrix of the i-th element.

[0091] Continue to refer to Figure 1 In step S140, the aggregated representation information of words is obtained by aggregating the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of words; the associated words of attribute words are determined among the above words, and the aggregated representation information of associated words is fused based on the adjacency matrix to obtain the final representation information of attribute words.

[0092] Aggregation can include operations such as addition, weighting, and concatenation. The contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of each word are aggregated separately to obtain the aggregated representation information of each word.

[0093] In one implementation, the above-mentioned aggregation of word contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information to obtain aggregated word representation information may include the following steps:

[0094] The contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of a word are concatenated and then processed using a gated loop unit to obtain the first intermediate aggregation information of the word.

[0095] After weighting the contextual representation information and syntactic dependency representation information of a word, it is concatenated with the attribute-aware distance representation information to obtain the second intermediate aggregation information of the word.

[0096] Based on the first and second intermediate aggregation information of the word, the aggregation representation information of the word is obtained.

[0097] For example, the first intermediate aggregation information can be obtained by the following formula:

[0098] h ssm =BiGRU([h bert h syn h dis (13)

[0099] The second intermediate aggregation information can be obtained using the following formula:

[0100]

[0101] The aggregation representation information of words can be obtained through the following formula:

[0102] h ssa =[h ssm h ssc (15)

[0103] Among them, h ssm The first intermediate aggregation information can be obtained by concatenating the three parts of information and then using a gated loop unit for context modeling. ssc The second intermediate aggregation information can be obtained by weighting the contextual representation information and syntactic dependency representation information of a word, and then concatenating it with the attribute-aware distance representation information. λ5 is a hyperparameter, and the symbol ";" indicates the concatenation operation. Concatenating the first and second intermediate aggregation information yields the aggregation representation information h of each word. ssa .

[0104] Associates of attribute words refer to words in the sequence to be processed that have a syntactic connection with the attribute word, such as words in the syntactic dependency tree that are directly or indirectly connected to the attribute word. The length of the connection path between an attribute word and its associated words can be set to not exceed a certain value (e.g., the association path length). Then, any word in the syntactic dependency tree that has a connection path with the attribute word, and whose path length does not exceed the association path length, can be used as an associated word of the attribute word.

[0105] In one implementation, reference Figure 5As shown, the process of determining the associated words of the attribute words in the above-mentioned words, and fusing the aggregated representation information of the associated words based on the adjacency matrix to obtain the final representation information of the attribute words, may include the following steps S510 and S520:

[0106] Step S510: Using words as nodes, form a graph corresponding to the sequence to be processed;

[0107] Step S520: Determine the first-level or multi-level related words of the attribute words in the graph, fuse the aggregated representation information of the highest-level related words based on the adjacency matrix and further fuse them level by level, and obtain the final representation information of the attribute words based on the fusion result.

[0108] The initial features of nodes in the graph can be derived from word aggregation representations. Edges in the graph can be generated based on the connections between words in the syntactic dependency tree. Words directly or indirectly connected to attribute words in the graph can be used as their associated words. For example, words directly connected to attribute words (i.e., adjacent words) can be found as first-level associated words, then words directly connected to first-level associated words can be found as second-level associated words, and so on. A maximum level f can be preset, and after finding k-level associated words, all associated words of the attribute word are obtained. The higher the level, the more distant the relationship with the attribute word. The highest-level associated word can be f-level associated words. First-level or multi-level fusion of associated words can be performed using graph convolution operations, and the final representation information of the attribute word can be obtained based on the fusion result.

[0109] For example, in the t-th semantic and syntactic-aware adjacency matrix and the l-th convolutional operation, the calculation formula for the graph convolution operation is as follows:

[0110]

[0111]

[0112] in, This represents the input features of the j-th word in the l-th layer of the t-th semantic and syntactic adjacency matrix. This represents the fused representation of the j-th word in the l-th layer of the t-th semantic and syntactic adjacency matrix. Let f(·) represent the attention weight between the i-th and j-th words in the t-th semantic and syntactic-aware adjacency matrix, and let f(·) be the average pooling operation. This represents the final word representation of the i-th word at layer l, which integrates multiple information elements. Thus, we obtain the final word representation of each word at layer l, which integrates multiple information elements. That is, the final representational information of attribute words.

[0113] In one implementation, each word in the attribute words is a word representation that incorporates multiple pieces of information. It can be done through a mask The non-attribute words are used to obtain the overall representation of the attribute words that incorporate multiple information, and then average pooling is used to obtain the overall representation of the attribute words. It can be considered as part of the final representation information of attribute words.

[0114] Continue to refer to Figure 1 In step S150, the final representation information of the attribute words is processed using a sentiment prediction model to obtain the sentiment prediction results of the attribute words.

[0115] For example, a sentiment prediction model can consist of a multilayer perceptron and an output layer (such as a softmax function layer). After the final representation information of the attribute words is input into the sentiment prediction model, its calculation process is as follows:

[0116]

[0117]

[0118] Among them, W E b E These are learnable model parameters. h represents the overall representation of attribute words that integrate multiple pieces of information. E This represents the overall representation of attribute words after the ReLU function has been applied, incorporating multiple pieces of information. This indicates the distribution of predicted sentiment polarity, such as whether the predicted attribute word's sentiment polarity is positive, negative, or neutral. This leads to a more accurate sentiment prediction result.

[0119] In one implementation, Figure 1 The processing can be implemented using a global model. (See reference) Figure 6As shown, the global model can include an embedding model, an adjacency processing model, a multi-information association model, and a sentiment prediction model. After inputting the sequence to be processed, syntactic dependency type, and attribute-aware distance into the model, it is first input into the embedding model, which includes a first embedding unit, a second embedding unit, and a third embedding unit. The first embedding unit processes the sequence to be processed to obtain the contextual representation information of the words, and obtains the initial representation information of the attribute words based on the contextual representation information. The second embedding unit processes the syntactic dependency type to obtain the syntactic dependency representation information of the words, and the third embedding unit processes the attribute-aware distance to obtain the attribute-aware distance representation information of the words. The contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words are input into the adjacency relation processing model. This model includes a syntactic awareness attention unit, a syntactic awareness self-attention unit, and a syntactic distance gating unit. The syntactic awareness attention unit processes these information to obtain a first association relation. The syntactic awareness self-attention unit processes these information to obtain a second association relation. The first and second association relations are then summed and processed by the syntactic distance gating unit to obtain the adjacency matrix. The same information is then input into a multi-information association model. This model includes a syntactic and semantic context modeling unit, a syntactic and semantic concatenation unit, a graph convolution operation unit, and an average pooling layer. The adjacency matrix is ​​then input into the graph convolution operation unit. The syntactic and semantic context modeling unit concatenates the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of a word, and then processes them using a gated recurrent unit to obtain the first intermediate aggregation information of the word. The syntactic and semantic concatenation unit then weights the contextual representation information and syntactic dependency representation information of the word and concatenates it with the attribute-aware distance representation information to obtain the second intermediate aggregation information of the word. The first intermediate aggregation information is then added to the second intermediate aggregation information to obtain the aggregated representation information of the word. The graph convolution operation unit then fuses the aggregated representation information of the associated words of the attribute word based on the adjacency matrix, and performs pooling processing by an average pooling layer to obtain the final representation information of the attribute word. Finally, the final representation information of the attribute word is input into the sentiment prediction model, which includes a multilayer perceptron and an output layer. After processing, the sentiment prediction result of the attribute word is finally output.

[0120] Exemplary embodiments of this disclosure also provide a language sentiment prediction apparatus. (See reference...) Figure 7 As shown, the language emotion prediction device 700 may include the following program modules:

[0121] The information acquisition module 710 is configured to acquire a sequence of words to be processed, which is formed by the statement to be processed and the attribute words in the statement to be processed, and to acquire the syntactic dependency type of the words in the sequence of words and the attribute-aware distance of the words. The attribute-aware distance of the words represents the syntactic distance between the words and the attribute words.

[0122] The embedding processing module 720 is configured to perform embedding processing on the sequence to be processed, the syntactic dependency type of the word, and the attribute-aware distance of the word to obtain the context representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word; and to obtain the initial representation information of the attribute word based on the context representation information of the word.

[0123] The adjacency processing module 730 is configured to use an attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words, and generate an adjacency matrix based on the processing results.

[0124] The multi-information aggregation module 740 is configured to aggregate word contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information to obtain aggregated word representation information; determine the associated words of the attribute words among the above words, and fuse the aggregated representation information of the associated words based on the adjacency matrix to obtain the final representation information of the attribute words.

[0125] The prediction processing module 750 is configured to process the final representation information of attribute words using a sentiment prediction model to obtain the sentiment prediction results of the attribute words.

[0126] In one implementation, the above-mentioned use of an attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words, and to generate an adjacency matrix based on the processing results, includes:

[0127] The syntactic awareness attention mechanism is used to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words in order to extract the first association relationship between words and attribute words.

[0128] The syntactic-aware self-attention mechanism is used to process the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of words in order to extract the second association relationship between words;

[0129] Generate an adjacency matrix based on the first and second association relationships.

[0130] In one implementation, the above-mentioned use of syntactic-aware attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of attribute words to extract the first association relationship between words and attribute words includes:

[0131] The initial representation information of the attribute words, the syntactic dependency representation information of the words, and the attribute perception distance representation information are aggregated and then processed by a gated loop unit to obtain the first intermediate representation information.

[0132] After aggregating the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of a word, the gated loop unit is used for processing to obtain the second intermediate representation information.

[0133] By integrating the first intermediate representation information and the second intermediate representation information, the first association relationship is obtained.

[0134] In one implementation, the above-mentioned processing of the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of words using a syntactic-aware self-attention mechanism to extract the second association relationship between words includes:

[0135] The initial representation information of attribute words, the syntactic dependency representation information of words, and the attribute-aware distance representation information are aggregated and then processed using a gated loop unit to obtain query representation information and key-value representation information, respectively.

[0136] By integrating query representation information and key value representation information, a second association relationship is obtained.

[0137] In one implementation, generating the adjacency matrix based on the first association relationship and the second association relationship includes:

[0138] The first and second association relationships are merged to obtain the first intermediate matrix;

[0139] The first intermediate matrix is ​​numerically adjusted based on the attribute-aware distance of words to obtain the second intermediate matrix;

[0140] Generate an adjacency matrix based on the second intermediate matrix.

[0141] In one implementation, the above-mentioned aggregation of word contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information yields aggregated word representation information, including:

[0142] The contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of a word are concatenated and then processed using a gated loop unit to obtain the first intermediate aggregation information of the word.

[0143] After weighting the contextual representation information and syntactic dependency representation information of a word, it is concatenated with the attribute-aware distance representation information to obtain the second intermediate aggregation information of the word.

[0144] Based on the first and second intermediate aggregation information of the word, the aggregation representation information of the word is obtained.

[0145] In one implementation, the process of determining the associated words of the attribute words from the aforementioned words, and fusing the aggregated representation information of the associated words based on the adjacency matrix to obtain the final representation information of the attribute words, includes:

[0146] Using words as nodes, a graph corresponding to the sequence to be processed is formed;

[0147] In the graph, the first-level or multi-level related words of the attribute words are identified. Based on the adjacency matrix, the aggregated representation information of the highest-level related words is fused and further fused level by level. The final representation information of the attribute words is obtained according to the fusion result.

[0148] The specific details of each part of the above-mentioned device have been described in detail in the method section of the implementation plan. For any undisclosed details, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0149] Exemplary embodiments of this disclosure also provide a computer-readable storage medium that can be implemented as a program product including program code, which, when run on an electronic device, causes the electronic device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. In an alternative embodiment, the program product can be implemented as a portable compact disc read-only memory (CD-ROM) including program code and can run on an electronic device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0150] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0151] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0152] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0153] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0154] Exemplary embodiments of this disclosure also provide an electronic device. The electronic device may include a processor and a memory. The memory stores executable instructions for the processor, such as program code. The processor executes the executable instructions to perform the methods of this exemplary embodiment.

[0155] The following is for reference. Figure 8 The electronic device is illustrated by way of a general-purpose computing device. It should be understood that... Figure 8 The electronic device 800 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0156] like Figure 8 As shown, the electronic device 800 may include: a processor 810, a memory 820, a bus 830, an I / O (input / output) interface 840, and a network adapter 850.

[0157] The memory 820 may include volatile memory, such as RAM 821 and cache unit 822, and may also include non-volatile memory, such as ROM 823. The memory 820 may also include one or more program units 824, including but not limited to: an operating system, one or more application programs, other program units, and program data. Each or some combination of these examples may include an implementation of a network environment. For example, program unit 824 may include the modules described above.

[0158] Bus 830 is used to connect different components of electronic device 800 and may include data bus, address bus and control bus.

[0159] Electronic device 800 can communicate with one or more external devices 900 (such as keyboard, mouse, external controller, etc.) through I / O interface 840.

[0160] Electronic device 800 can communicate with one or more networks via network adapter 850. For example, network adapter 850 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication. Network adapter 850 can communicate with other units of electronic device 800 via bus 830.

[0161] although Figure 8 Other hardware and / or software units may also be provided in the electronic device 800, including but not limited to: display, microcode, device driver, redundant processor, external disk drive array, RAID system, tape drive and data backup storage system, etc.

[0162] It should be noted that although several units or modules of the device for performing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more units or modules described above can be embodied in one unit or module. Conversely, the features and functions of one unit or module described above can be further divided and embodied by multiple units or modules.

[0163] Those skilled in the art will understand that various aspects of this disclosure can be implemented as systems, methods, or program products. Therefore, various aspects of this disclosure can be embodied in entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuit,” “unit,” or “system.” Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0164] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is defined only by the appended claims.

Claims

1. A language sentiment prediction method, characterized in that, include: Obtain the processing sequence formed by the statement to be processed and the attribute words in the statement to be processed; obtain the syntactic dependency type of the words in the processing sequence and the attribute-aware distance of the words; the attribute-aware distance of the words represents the syntactic distance between the words and the attribute words. Embedding processing is performed on the sequence to be processed, the syntactic dependency type of the word, and the attribute-aware distance of the word to obtain the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word; the initial representation information of the attribute word is obtained based on the contextual representation information of the word. The attention mechanism is used to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words, and an adjacency matrix is ​​generated based on the processing results. By aggregating the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word, the aggregated representation information of the word is obtained. The associated words of the attribute word are determined in the word, and the aggregated representation information of the associated words is fused based on the adjacency matrix to obtain the final representation information of the attribute word; The final representation information of the attribute words is processed using a sentiment prediction model to obtain the sentiment prediction results of the attribute words. The sentiment prediction model includes a multilayer perceptron and an output layer; The step of using an attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words, and generating an adjacency matrix based on the processing results, includes: The syntactic awareness attention mechanism is used to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words to extract the first association relationship between the words and the attribute words. The contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the words are processed using a syntactic-aware self-attention mechanism to extract the second association relationship between the words. Generate an adjacency matrix based on the first association relationship and the second association relationship.

2. The method according to claim 1, characterized in that, The process of using a syntactic-aware attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words to extract the first association relationship between the words and the attribute words includes: The initial representation information of the attribute word, the syntactic dependency representation information of the word, and the attribute-aware distance representation information are aggregated and processed by a gated loop unit to obtain the first intermediate representation information. The contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word are aggregated and processed using a gated loop unit to obtain the second intermediate representation information. The first association relationship is obtained by fusing the first intermediate representation information and the second intermediate representation information.

3. The method according to claim 1, characterized in that, The process of using a syntactic-aware self-attention mechanism to process the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the words to extract the second association relationship between the words includes: The initial representation information of the attribute words, the syntactic dependency representation information of the words, and the attribute-aware distance representation information are aggregated and then processed using a gated loop unit to obtain query representation information and key-value representation information, respectively. The second association relationship is obtained by integrating the query representation information and the key value representation information.

4. The method according to claim 1, characterized in that, The step of generating an adjacency matrix based on the first association relationship and the second association relationship includes: The first association relationship and the second association relationship are merged to obtain the first intermediate matrix; The first intermediate matrix is ​​numerically adjusted based on the attribute perception distance of the words to obtain the second intermediate matrix; The adjacency matrix is ​​generated based on the second intermediate matrix.

5. The method according to claim 1, characterized in that, The aggregation of the word's contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information to obtain the aggregated representation information of the word includes: The contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word are concatenated and then processed using a gated loop unit to obtain the first intermediate aggregation information of the word. The contextual representation information and syntactic dependency representation information of the word are weighted and then concatenated with the attribute-aware distance representation information to obtain the second intermediate aggregation information of the word. Based on the first intermediate aggregation information and the second intermediate aggregation information of the word, the aggregation representation information of the word is obtained.

6. The method according to claim 1, characterized in that, The step of determining the associated words of the attribute word in the word, and fusing the aggregated representation information of the associated words based on the adjacency matrix to obtain the final representation information of the attribute word includes: Using the words as nodes, a graph corresponding to the sequence to be processed is formed; In the graph, the first-level or multi-level related words of the attribute word are determined. Based on the adjacency matrix, the aggregated representation information of the highest-level related words is fused and further fused level by level. The final representation information of the attribute word is obtained according to the fusion result.

7. A language emotion prediction device, characterized in that, include: The information acquisition module is configured to acquire a sequence to be processed formed by a statement to be processed and attribute words in the statement to be processed, acquire the syntactic dependency type of the words in the sequence to be processed and the attribute-aware distance of the words, wherein the attribute-aware distance of the words represents the syntactic distance between the words and the attribute words. The embedding processing module is configured to perform embedding processing on the sequence to be processed, the syntactic dependency type of the word, and the attribute-aware distance of the word to obtain the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word; and to obtain the initial representation information of the attribute word based on the contextual representation information of the word. The adjacency processing module is configured to use an attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words, and generate an adjacency matrix based on the processing results. The multi-information aggregation module is configured to obtain the aggregated representation information of the word by aggregating the contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the word. The associated words of the attribute word are determined in the word, and the aggregated representation information of the associated words is fused based on the adjacency matrix to obtain the final representation information of the attribute word; The prediction processing module is configured to process the final representation information of the attribute words using a sentiment prediction model to obtain the sentiment prediction result of the attribute words. The sentiment prediction model includes a multilayer perceptron and an output layer; The step of using an attention mechanism to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words, and generating an adjacency matrix based on the processing results, includes: The syntactic awareness attention mechanism is used to process the contextual representation information, syntactic dependency representation information, attribute-aware distance representation information, and initial representation information of the attribute words to extract the first association relationship between the words and the attribute words. The contextual representation information, syntactic dependency representation information, and attribute-aware distance representation information of the words are processed using a syntactic-aware self-attention mechanism to extract the second association relationship between the words. Generate an adjacency matrix based on the first association relationship and the second association relationship.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 6 by executing the executable instructions.

Images