Hierarchical topic identification method and device for user opinion stream data

Abstract

The present invention provides a hierarchical topic identification method and device for user opinion stream data. The method comprises: pre-processing the user opinion stream data to obtain an initial text feature representation, inputting the initial text feature representation into the encoder of the Transformer to obtain a key feature matrix and a value feature matrix; inputting the label tree corresponding to the user opinion data into a directed graph neural network to obtain an initial label matrix, inputting the initial label matrix into the decoder module of the Transformer to obtain a query feature matrix; jointly inputting the key feature matrix, the value feature matrix and the query feature matrix into the Transformer decoder to obtain a label matrix after feature interaction, and then passing the label matrix through a fully connected layer to obtain the label category probability of the user opinion stream data, and taking the label with a probability value greater than a set threshold as the hierarchical topic category of the user opinion stream data. The present invention models the context dependency relationship of the label hierarchical structure information, which can effectively improve the performance of hierarchical multi-label classification in user opinion stream data.

Description

Technical Field

[0001] The present invention relates to the technical field of data stream identification, and in particular to a hierarchical topic identification method and device for user opinion stream data. Background Art

[0002] With the rapid development of the national economy and the continuous strengthening of China's comprehensive national strength, the public's demand for higher-quality civil aviation services continues to rise. The quality of air transport services directly impacts passenger experience and perceptions, which in turn affects people's lives and even international competitiveness. The widespread adoption of social networks and multimedia technologies has greatly expanded the platforms for public opinion exchange. To improve the quality of my country's air transport services, it is necessary to continuously expand passenger opinion collection platforms, including the 12326 call system and the civil aviation service quality supervision platform. Every day, every moment, passengers' opinions on flight regularity, ticketing, baggage transportation, and catering services are fed into the network. This massive amount of passenger opinion stream data poses a significant challenge for accurately understanding public sentiment in civil aviation services. Rapidly and accurately identifying topic categories from this massive amount of passenger opinion stream data and automatically categorizing passenger opinions into specific categories can significantly reduce the labor costs of passenger service personnel and significantly improve the efficiency of passenger opinion analysis and processing. Therefore, how to automatically identify topics in passenger opinion stream data within the civil aviation service sector is a pressing issue.

[0003] Given a hierarchical topic labeling system, identifying hierarchical topics in passenger opinion stream data from the civil aviation service sector is a hierarchical multi-label classification task. Hierarchical multi-label classification, a special subtask of multi-label text classification, has been a hot topic of academic research and is applicable to many application scenarios. For example, five-level hierarchical topic classification of passenger opinion information in the civil aviation service sector, labeling unclassified news with hierarchical categories according to the hierarchical classification system of news portals, and labeling unclassified products according to the hierarchical classification system of product catalogs on shopping websites are all examples of hierarchical multi-label classification tasks. The characteristic of hierarchical multi-label classification tasks is the presence of a hierarchical structure in the labels of the dataset being studied. The goal is to predict multiple labels for text given a hierarchical labeling system.

[0004] Currently, existing hierarchical multi-label topic identification methods include flat, local, and global methods. Flat methods build multiple binary classifiers equal to the total number of labels, independently predicting each label while ignoring the hierarchical relationships between labels and predicting only the fine-grained labels at the highest level. This type of method ignores a large amount of information and has low classification performance. Local methods construct multiple local classifiers for each hierarchical category and rely on the category hierarchy to establish connections between local classifiers at each level. For example, a child classifier inherits the model parameters of the parent classifier for initialization and fine-tuning. This type of method has a large number of model parameters and cannot fully model the dependencies between classes by relying solely on parameter initialization. If an error occurs in the parent local classifier, the error will be passed on to the child classifier, which can easily lead to bias. Global methods build a classifier that directly encodes the complete label structure information and predicts all labels simultaneously. Most methods attempt to continuously learn the hierarchical structure of labels using strategies such as graph learning, reinforcement learning, capsule networks, and structural encoding. They then incorporate this multi-label type hierarchy information into the feature vector of the text to be predicted through semantic fusion or semantic matching strategies, thereby performing multi-label type prediction. This type of method is currently the mainstream method due to its good performance. Although these methods have achieved certain results, the performance of most methods is still relatively low (the accuracy rate is around 79%), and they are unable to fully learn the hierarchical structure information between labels and effectively integrate label and text information.

[0005] The disadvantages of the above-mentioned hierarchical multi-label topic identification method in the prior art include: the flat method ignores a lot of information and has low classification performance.

[0006] The local method model has a large number of parameters, and relying solely on parameter initialization cannot fully model the dependencies between classes. If an error occurs in the local classifier at the parent level, the error information will be passed to the child level, which can easily lead to deviations.

[0007] The performance of the global method is still low, and it cannot fully learn the hierarchical information between labels and cannot fully integrate label and text information. Summary of the Invention

[0008] The embodiments of the present invention provide a hierarchical topic identification method and apparatus for user opinion stream data, which are used to solve the problem that current technical solutions cannot fully learn the hierarchical structure information between tags and cannot fully integrate tags and text information.

[0009] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0010] According to one aspect of the present invention, a hierarchical topic identification method for user opinion stream data is provided, comprising:

[0011] Inputting user opinion stream data into a pre-trained word vector model, the pre-trained word vector model outputs a word vector representation;

[0012] Input the word vector representation into the bidirectional recurrent neural network Bi-GRU to obtain the initial text feature representation corresponding to the temporal dependent semantic information;

[0013] Input the initial text feature representation into the encoder module of Transformer to obtain a key feature matrix and a value feature matrix representing text feature information;

[0014] Constructing a label tree based on the five-level hierarchical label system corresponding to the user opinion data, and inputting the label tree into a directed graph convolutional neural network to obtain an initial label matrix representing the hierarchical semantic structure information;

[0015] Input the initial label matrix into the first multi-head attention module of the Transformer decoder,

[0016] Get the query feature matrix;

[0017] The key feature matrix, value feature matrix, and query feature matrix are jointly input into the second multi-head attention module of the Transformer decoder to obtain the label matrix after feature interaction;

[0018] The label matrix is ​​then passed through the fully connected layer to obtain the label category probability of the user opinion stream data, and the label with a probability value greater than the set threshold is taken as the hierarchical topic category of the user opinion stream data.

[0019] Preferably, the step of inputting the initial text feature representation into the encoder module of the Transformer to obtain a key feature matrix and a value feature matrix representing text feature information includes:

[0020] The initial text feature representation is input into the encoder module of the Transformer, and the multi-head attention mechanism is used to fuse the information of different subspaces. After two residual connections and layer normalization operations, the encoder module is used to model the long-range dependency relationship of the text features to obtain text feature information. After linear mapping, a new key feature matrix and value feature matrix are output;

[0021] In the Transformer modeling process, the single-head attention mechanism is defined as follows:

[0022]

[0023] Among them, Q, K and V are feature matrices obtained by packaging multiple input text features together after linear mapping, and q is the preset scaling factor.

[0024] Based on the single-head attention mechanism, the multi-head attention mechanism is defined as follows:

[0025] H Z =Attention(Q Z ,K Z ,V Z ),z=1,2,...,Z

[0026] X=Concat(H1,H2,...,H Z )W

[0027] Where Z is the number of attention heads, Concat represents the concatenation operation, W represents the trainable linear transformation matrix, and X is the text feature representation obtained after fusing multiple attention heads.

[0028] After linear mapping, X is packed together to obtain new key feature matrices and value feature matrices.

[0029] Preferably, the step of constructing a tag tree based on a five-level hierarchical tag system corresponding to user opinion data includes:

[0030] A five-level hierarchical labeling system for user opinion stream data is constructed. DGCN is used to perform orderly relationship modeling on the five-level hierarchical labeling system of user opinion stream data, and a five-level label tree is established for user opinion stream data discrimination. The latter level in the five-level label tree is a detailed classification of the previous level. There is a master-slave relationship between each level. The final leaf node represents the complete classification path from the first level classification to the final level classification. The categories of each level can be inferred based on the categories of the leaf nodes.

[0031] Preferably, the step of inputting the label tree into a directed graph convolutional neural network to obtain an initial label matrix representing hierarchical semantic structure information comprises:

[0032] The label tree is input into a two-layer directed graph convolutional neural network. After the label text in the label tree is embedded vectorized, the hierarchical relationship between the labels is modeled using a directed graph convolutional neural network. The formula is:

[0033]

[0034] in, A is the directed adjacency matrix between labels derived from the label tree, I is the identity matrix, for The degree matrix, H (l) Represents the label feature matrix of the lth layer, where l is 0 and 1 respectively, indicating that it has been modeled through two graph neural networks. When l is 0, H (0)It indicates that the initial moment is the label feature matrix after the embedded vectorization processing of the label text in the label tree. After passing through the two-layer directed graph convolutional neural network, the initial label matrix containing a finer-grained hierarchical structure and semantic information is obtained.

[0035] Preferably, the initial label matrix is ​​input into the first multi-head attention module of the Transformer decoder to obtain the query feature matrix, including:

[0036] The initial label matrix is ​​input into the first multi-head attention module of the Transformer decoder. Using the multi-head attention mechanism, the Transformer decoder module obtains label feature information after a residual connection and layer normalization operation. After linear mapping, it outputs a new query feature matrix.

[0037] In the Transformer modeling process, the single-head attention mechanism is defined as follows:

[0038]

[0039] Among them, Q, K and V are feature matrices obtained by packaging multiple input label matrices together after linear mapping, and q is a preset scaling factor;

[0040] Based on the single-head attention mechanism, the multi-head attention mechanism is defined as follows:

[0041] H Z =Attention(Q Z ,K Z ,V Z ),z=1,2,...,Z

[0042] X=Concat(H1,H2,...,H Z )W

[0043] Where Z is the number of attention heads, Concat represents the concatenation operation, W represents the trainable linear transformation matrix, and X is the label feature information obtained after fusing multiple attention heads.

[0044] After linear mapping, X is packaged together to obtain a new query feature matrix.

[0045] Preferably, the key feature matrix, the value feature matrix, and the query feature matrix are jointly input into the second multi-head attention module of the Transformer decoder to obtain the label matrix after feature interaction, including:

[0046] The key feature matrix and value feature matrix representing the text features are used as the two inputs of the second multi-head attention of the Transformer decoder, and the query feature matrix is ​​used as the other input of the second multi-head attention of the Transformer decoder. The key feature matrix, value feature matrix and query feature matrix are modeled with multi-layer self-attention interaction, so that each label automatically pays attention to the corresponding text features. The label features are refined through the Feed Forward linear layer to generate F label ∈R N×d , N is the number of leaf node labels, d is the feature dimension, and finally the d-dimensional features are mapped to 1-dimensional through the fully connected layer, thereby generating N values ​​and obtaining the label matrix after feature interaction for multi-label classification.

[0047] Preferably, the step of passing the label matrix through a fully connected layer to obtain the label category probabilities of the user opinion stream data, and taking the labels with probability values ​​greater than a set threshold as the hierarchical topic categories of the user opinion stream data, includes:

[0048] The label matrix after the feature interaction is passed through the fully connected layer to obtain an N-dimensional feature vector, which is mapped to a probability score p of [0,1] through the sigmoid layer. ij , the probability score p ij The probability of user opinion stream data text i being classified as label j is expressed, and the label category with a probability value greater than a set threshold is determined as the label category of the user opinion stream data;

[0049] During training, the binary cross entropy loss function is used for model optimization, and the formula is:

[0050]

[0051] Among them, y ij It represents the manually labeled category label. Its value is 1, indicating that it belongs to the category; its value is 0, indicating that it does not belong to the category. ij It represents the category probability predicted by the model, and its value belongs to [0,1]. M and N represent the total number of user opinion stream data texts and the total number of category labels, respectively. L represents the loss function established for the category probability output by the entire model.

[0052] According to one aspect of the present invention, a device for quickly identifying hierarchical topics in user opinion stream data is provided, comprising:

[0053] A word vector representation acquisition module is used to input user opinion stream data into a pre-trained word vector model, and the pre-trained word vector model outputs a word vector representation;

[0054] An initial text feature representation acquisition module is used to input the word vector representation into a bidirectional recurrent neural network Bi-GRU to obtain an initial text feature representation corresponding to temporal dependent semantic information;

[0055] A key and value feature matrix acquisition module is used to input the initial text feature representation into the encoder module of the Transformer to obtain a key feature matrix and a value feature matrix representing text feature information;

[0056] An initial label matrix acquisition module is used to construct a label tree based on the five-level hierarchical label system corresponding to the user opinion data, and input the label tree into a directed graph convolutional neural network to obtain an initial label matrix representing the hierarchical semantic structure information;

[0057] The query feature matrix acquisition module is used to input the initial label matrix into the first multi-head attention module of the Transformer decoder to obtain the query feature matrix;

[0058] The label matrix acquisition module after feature interaction is used to jointly input the key feature matrix, value feature matrix and query feature matrix into the second multi-head attention module of the Transformer decoder to obtain the label matrix after feature interaction;

[0059] The user opinion stream data label category determination module is used to pass the label matrix through the fully connected layer to obtain the label category probability of the user opinion stream data, and take the label with a probability value greater than the set threshold as the hierarchical topic category of the user opinion stream data.

[0060] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the embodiments of the present invention aim to use a method based on transformer combined with directed graph neural networks to model contextual dependencies of label hierarchical information, and fully consider the rich dependencies between text and labels, so as to effectively improve the performance of hierarchical multi-label classification in user opinion stream data.

[0061] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1This is a diagram showing the overall implementation framework of a hierarchical topic identification method for user opinion stream data provided by an embodiment of the present invention;

[0064] Figure 2 A processing flow chart of a hierarchical topic identification method for user opinion stream data provided by an embodiment of the present invention;

[0065] Figure 3 An embodiment of the present invention also provides a structural diagram of a hierarchical topic identification device for user opinion stream data. DETAILED DESCRIPTION

[0066] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0067] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0068] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0069] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0070] Example 1

[0071] The present invention adopts a codec structure and utilizes a transformer-based hierarchical multi-label classification method to effectively model user opinion stream data and achieve rapid identification of hierarchical topics. The overall implementation framework diagram of a hierarchical topic identification method for user opinion stream data provided by the embodiment of the present invention is as follows: Figure 1 As shown, the processing flow is as follows Figure 2 As shown, the processing steps include the following:

[0072] Step S10: input the user opinion stream data into a pre-trained word vector model, and the pre-trained word vector model outputs a word vector representation.

[0073] The Word2Vec pre-training model uses the Huffman Tree-based CBOW model and the Negative Sampling-based Skip-gram model to implement batch natural language word representation computation. It uses the learned word vectors to calculate the semantic similarity and relevance of target words, as well as the semantic angle between word vectors. It also measures the time required for batch representation computation under different increments. This model is an unsupervised word representation learning model and has become a mainstream deep learning representation learning method since its inception.

[0074] Step S20: Figure 1 In the left half of the sentence, the word vector representation obtained in step S10 is input into the bidirectional recurrent neural network Bi-GRU to learn the temporal dependency semantic information and obtain the initial text feature representation corresponding to the temporal dependency semantic information, namely Text Embeddings.

[0075] The bidirectional recurrent neural network Bi-GRU model is a recurrent neural network model that performs forward and reverse context modeling by feeding the text representation into the forward and reverse recurrent neural networks GRU respectively, and obtains the text representation containing context features through feature concatenation.

[0076] Step S30: Figure 1 In the right half of the Transformer, the text embeddings obtained in step S20 are input into the Transformer encoder module. A multi-head attention mechanism is used to address long-range text dependencies and fuse information from different subspaces. After two residual connections and layer normalization operations, the Transformer encoder module uses the encoder module to model long-range dependencies between text features, obtaining richer text feature information and outputting key and value feature matrices.

[0077] The core modules of the Transformer encoder are a multi-head self-attention module and a feedforward neural network. The former is used to model the self-attention interactions of input features in multiple feature spaces, while the latter is used to refine the features of the former. The Transformer decoder consists of three parts. The first part performs self-attention modeling on the decoder's input features and outputs a query feature matrix. The second part performs cross-attention modeling on the query feature matrix and the key and value feature matrices output from the encoder. The query matrix incorporates contextual information from the encoder and outputs the resulting interacted features. The third part feeds the interacted query features into the feedforward neural network to output the refined features.

[0078] The following steps S40 , S50 and S60 may be performed in parallel with the above steps S10 , S20 and S30 .

[0079] Step S40: Construct a tag tree using the five-level hierarchical tag system corresponding to the user opinion data. A partial example of the five-level hierarchical tag system for civil aviation passenger user opinion stream data is shown in the following table.

[0080]

[0081]

[0082] like Figure 2 As shown in the figure, the DGCN modeling of the five-level label classification system is described. Based on years of operational experience, a five-level label tree is established for user opinion flow data discrimination. The latter level is a detailed classification of the previous level. Therefore, there is a master-slave relationship between the levels. The final leaf node represents the complete classification path from the first level classification to the final level classification. Therefore, given the category of the leaf node, the category of each level can be inferred.

[0083] Step S50: Input the label tree into a two-layer directed graph convolutional neural network (DGCN). Using DGCN, the constructed five-level hierarchical label system of the user opinion stream data is sequentially modeled, extracting more fine-grained hierarchical semantic structure information and obtaining a representation of the label tree leaf node labels, i.e., the label matrix Label Embeddings. This method uses leaf node embeddings as the basis for text classification.

[0084] After embedding the label text in the label tree into vector form, the directed graph convolutional neural network (DGCN) is used to model the hierarchical relationship between labels. The formula is:

[0085]

[0086] in, A is the directed adjacency matrix between labels derived from the label tree, I is the identity matrix, for The degree matrix, H (l) Represents the label feature matrix of the lth layer, where l is 0 and 1 respectively, indicating that it has been modeled through two graph neural networks. When l is 0, H (0) H represents the label feature matrix after the embedded vectorization of the label text in the label tree at the initial moment, (1) Represents the label feature matrix after one graph neural network modeling, H (2) The label feature matrix after two graph neural network modeling is shown in Figure 2. After two layers of DGCN transmission, the initial label matrix Label embeddings containing finer-grained hierarchical structure and semantic information is finally obtained.

[0087] Step S60: Input the initial label matrix Label Embeddings obtained in step S50 into the decoder module of the Transformer. Using the multi-head attention mechanism, the decoder module of the Transformer obtains richer label feature information after a residual connection and layer normalization operation.

[0088] In the Transformer modeling process, the single-head attention mechanism is defined as follows:

[0089]

[0090] Among them, Q, K and V are feature matrices obtained by packaging multiple input embeddings together after linear mapping, and q is the preset scaling factor.

[0091] On this basis, the multi-head attention mechanism is defined as follows:

[0092] H Z =Attention(Q Z ,K Z ,V Z ),z=1,2,...,Z

[0093] X=Concat(H1,H2,...,H Z )W

[0094] Where Z is the number of attention heads, Concat represents the concatenation operation, W represents the trainable linear transformation matrix, and X is the embedding representation obtained by fusing multiple attention heads.

[0095] Step S70: The key feature matrix and the value feature matrix representing the text features are used as two inputs of the second multi-head attention of the Transformer decoder, and the query feature matrix is ​​used as another input of the second multi-head attention of the Transformer decoder. The key feature matrix, the value feature matrix and the query feature matrix are subjected to multi-layer self-attention interaction modeling, so that each label automatically pays attention to the corresponding text features, and the label features are refined through the Feed Forward linear layer to generate F label ∈R N×d , N is the number of leaf node labels, d is the feature dimension, and finally the d-dimensional features are mapped to 1-dimensional through the fully connected layer, thereby generating N values ​​and obtaining the label matrix after feature interaction for multi-label classification.

[0096] Step S80: The Output Embeddings obtained in step S70 are passed through a fully connected layer to obtain an N-dimensional feature vector, and then passed through a Sigmoid layer to obtain the probability value of each label category. The label category with a probability value greater than the threshold is the obtained label category.

[0097] Mapped to [0,1] probability score p through sigmoid layer ij , represents the probability that the user opinion stream data text i is classified as label j. The larger the value, the greater the possibility.

[0098] During training, the binary cross entropy loss function is used for model optimization, and the formula is:

[0099]

[0100] This formula is to train and optimize the model, where y ij It represents the manually labeled category label, with a value of 1 indicating that it belongs to the category; a value of 0 indicates that it does not belong to the category. ij represents the class probability predicted by the model, with a value between 0 and 1. M and N represent the total number of texts in the user opinion stream and the total number of class labels, respectively. L represents the loss function established for the class probability output by the entire model, thereby optimizing the parameters of the entire model.

[0101] Example 2

[0102] The embodiment of the present invention also provides a hierarchical topic identification device for user opinion stream data, the structure of which is shown in FIG. Figure 3 As shown, it includes the following modules:

[0103] A word vector representation acquisition module is used to input user opinion stream data into a pre-trained word vector model, and the pre-trained word vector model outputs a word vector representation;

[0104] An initial text feature representation acquisition module is used to input the word vector representation into a bidirectional recurrent neural network Bi-GRU to obtain an initial text feature representation corresponding to temporal dependent semantic information;

[0105] A key and value feature matrix acquisition module is used to input the initial text feature representation into the encoder module of the Transformer to obtain a key feature matrix and a value feature matrix representing text feature information;

[0106] An initial label matrix acquisition module is used to construct a label tree based on the five-level hierarchical label system corresponding to the user opinion data, and input the label tree into a directed graph convolutional neural network to obtain an initial label matrix representing the hierarchical semantic structure information;

[0107] The query feature matrix acquisition module is used to input the initial label matrix into the first multi-head attention module of the Transformer decoder to obtain the query feature matrix;

[0108] The label matrix acquisition module after feature interaction is used to jointly input the key feature matrix, value feature matrix and query feature matrix into the second multi-head attention module of the Transformer decoder to obtain the label matrix after feature interaction;

[0109] The user opinion stream data label category determination module is used to pass the label matrix through the fully connected layer to obtain the label category probability of the user opinion stream data, and take the label with a probability value greater than the set threshold as the hierarchical topic category of the user opinion stream data.

[0110] The specific process of performing hierarchical topic identification for user opinion stream data using the apparatus of the embodiment of the present invention is similar to that of the aforementioned method embodiment and will not be described in detail here.

[0111] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the hierarchical topic identification method for user opinion stream data as described above is implemented.

[0112] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed on one or more processors, is used to implement the above-mentioned hierarchical topic identification method for user opinion stream data.

[0113] An embodiment of the present invention also provides an electronic device, comprising: a processor, a memory, and a computer program, wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the hierarchical topic identification method for user opinion stream data as described above.

[0114] In summary, the embodiment of the present invention utilizes a multi-head attention mechanism to learn text feature representation, solves the problem of long-distance text dependency, and fuses information from different subspaces. After two residual connections and layer normalization operations, richer text feature information is obtained.

[0115] By using a directed graph convolutional neural network to fully learn the intermediate node label information and structural dependency information of the label tree, the present invention ensures that the representation of leaf node labels contains the semantic information of all hierarchical labels along its path. By utilizing a multi-head attention mechanism to learn label feature representations, the present invention obtains richer label feature information after a single residual connection and layer normalization operation. This method can more effectively fuse the semantic information of text and labels, and the resulting fused feature representation is more conducive to multi-label classification tasks.

[0116] The present invention adopts DGCN to carry out orderly relationship modeling on the five-level hierarchical label system of constructed user opinion stream data, and uses the embedding of leaf nodes as the basis for text classification. It has better scalability and can be expanded to more levels of classification system, and better models the master-slave relationship between labels at all levels.

[0117] The main body of the present invention adopts the Transformer structure to model the dependency relationship between user opinion stream data and its labels. The encoder part performs rich semantic information mining on the text stream data, and the decoder part naturally performs context modeling on the dependency relationship between the text stream data and the label data, so that each label can automatically focus on the most important text features to it, thereby making the predicted probability value more robust and more accurate.

[0118] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0119] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0120] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0121] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A hierarchical topic identification method for user opinion stream data, characterized in that: include: Inputting user opinion stream data into a pre-trained word vector model, the pre-trained word vector model outputs a word vector representation; Input the word vector representation into the bidirectional recurrent neural network Bi-GRU to obtain the initial text feature representation corresponding to the temporal dependent semantic information; Input the initial text feature representation into the encoder module of Transformer to obtain a key feature matrix and a value feature matrix representing text feature information; Constructing a label tree based on the five-level hierarchical label system corresponding to the user opinion data, and inputting the label tree into a directed graph convolutional neural network to obtain an initial label matrix representing the hierarchical semantic structure information; Input the initial label matrix into the first multi-head attention module of the Transformer decoder to obtain the query feature matrix; The key feature matrix, value feature matrix, and query feature matrix are jointly input into the second multi-head attention module of the Transformer decoder to obtain the label matrix after feature interaction; The label matrix is ​​then passed through the fully connected layer to obtain the label category probability of the user opinion stream data, and the label with a probability value greater than the set threshold is taken as the hierarchical topic category of the user opinion stream data.

2. The method according to claim 1, characterized in that The step of inputting the initial text feature representation into the encoder module of the Transformer to obtain a key feature matrix and a value feature matrix representing text feature information includes: The initial text feature representation is input into the encoder module of the Transformer, and the multi-head attention mechanism is used to fuse the information of different subspaces. After two residual connections and layer normalization operations, the encoder module is used to model the long-range dependency relationship of the text features to obtain text feature information. After linear mapping, a new key feature matrix and value feature matrix are output; In the Transformer modeling process, the single-head attention mechanism is defined as follows: Among them, Q, K and V are feature matrices obtained by packaging multiple input text features together after linear mapping, and q is the preset scaling factor. Based on the single-head attention mechanism, the multi-head attention mechanism is defined as follows: H Z =Attention(Q Z ,K Z ,V Z ),z=1,2,...,Z X=Concat(H1,H2,...,H Z )W Where Z is the number of attention heads, Concat represents the concatenation operation, W represents the trainable linear transformation matrix, and X is the text feature representation obtained after fusing multiple attention heads. After linear mapping, X is packed together to obtain new key feature matrices and value feature matrices.

3. The method according to claim 1, characterized in that The tag tree is constructed based on the five-level hierarchical tag system corresponding to the user opinion data, including: A five-level hierarchical labeling system for user opinion stream data is constructed. DGCN is used to perform orderly relationship modeling on the five-level hierarchical labeling system of user opinion stream data, and a five-level label tree is established for user opinion stream data discrimination. The latter level in the five-level label tree is a detailed classification of the previous level. There is a master-slave relationship between each level. The final leaf node represents the complete classification path from the first level classification to the final level classification. The categories of each level can be inferred based on the categories of the leaf nodes.

4. The method according to claim 3, characterized in that The step of inputting the label tree into a directed graph convolutional neural network to obtain an initial label matrix representing hierarchical semantic structure information includes: The label tree is input into a two-layer directed graph convolutional neural network. After the label text in the label tree is embedded vectorized, the hierarchical relationship between the labels is modeled using a directed graph convolutional neural network. The formula is: in, A is the directed adjacency matrix between labels derived from the label tree, I is the identity matrix, for The degree matrix, H (l) Represents the label feature matrix of the lth layer, where l is 0 and 1 respectively, indicating that it has been modeled through two graph neural networks. When l is 0, H (0) It indicates that the initial moment is the label feature matrix after the embedded vectorization processing of the label text in the label tree. After passing through the two-layer directed graph convolutional neural network, the initial label matrix containing a finer-grained hierarchical structure and semantic information is obtained.

5. The method according to claim 4, characterized in that The initial label matrix is ​​input into the first multi-head attention module of the Transformer decoder to obtain the query feature matrix, including: The initial label matrix is ​​input into the first multi-head attention module of the Transformer decoder. Using the multi-head attention mechanism, the Transformer decoder module obtains label feature information after a residual connection and layer normalization operation. After linear mapping, it outputs a new query feature matrix. In the Transformer modeling process, the single-head attention mechanism is defined as follows: Among them, Q, K and V are feature matrices obtained by packaging multiple input label matrices together after linear mapping, and q is a preset scaling factor; Based on the single-head attention mechanism, the multi-head attention mechanism is defined as follows: H Z =Attention(Q Z ,K Z ,V Z ),z=1,2,...,Z X=Concat(H1,H2,...,H Z )W Where Z is the number of attention heads, Concat represents the concatenation operation, W represents the trainable linear transformation matrix, and X is the label feature information obtained after fusing multiple attention heads. After linear mapping, X is packaged together to obtain a new query feature matrix.

6. The method according to claim 5, characterized in that The key feature matrix, value feature matrix, and query feature matrix are jointly input into the second multi-head attention module of the Transformer decoder to obtain the label matrix after feature interaction, including: The key feature matrix and value feature matrix representing the text features are used as the two inputs of the second multi-head attention of the Transformer decoder, and the query feature matrix is ​​used as the other input of the second multi-head attention of the Transformer decoder. The key feature matrix, value feature matrix and query feature matrix are modeled with multi-layer self-attention interaction, so that each label automatically pays attention to the corresponding text features. The label features are refined through the Feed Forward linear layer to generate F label ∈R N×d , N is the number of leaf node labels, d is the feature dimension, and finally the d-dimensional features are mapped to 1-dimensional through the fully connected layer, thereby generating N values ​​and obtaining the label matrix after feature interaction for multi-label classification.

7. The method according to claim 6, characterized in that The process of passing the label matrix through the fully connected layer to obtain the label category probabilities of the user opinion stream data and taking the labels with probability values ​​greater than a set threshold as the hierarchical topic categories of the user opinion stream data includes: The label matrix after the feature interaction is passed through the fully connected layer to obtain an N-dimensional feature vector, which is mapped to a probability score p of [0,1] through the sigmoid layer. ij , the probability score p ij The probability of user opinion stream data text i being classified as label j is expressed, and the label category with a probability value greater than a set threshold is determined as the label category of the user opinion stream data; During training, the binary cross entropy loss function is used for model optimization, and the formula is: Among them, y ij It represents the manually labeled category label. Its value is 1, indicating that it belongs to the category; its value is 0, indicating that it does not belong to the category. ij It represents the category probability predicted by the model, and its value belongs to [0,1]. M and N represent the total number of user opinion stream data texts and the total number of category labels, respectively. L represents the loss function established for the category probability output by the entire model.

8. A device for quickly identifying hierarchical topics based on user opinion stream data, characterized in that: include: A word vector representation acquisition module is used to input user opinion stream data into a pre-trained word vector model, and the pre-trained word vector model outputs a word vector representation; An initial text feature representation acquisition module is used to input the word vector representation into a bidirectional recurrent neural network Bi-GRU to obtain an initial text feature representation corresponding to temporal dependent semantic information; A key and value feature matrix acquisition module is used to input the initial text feature representation into the encoder module of the Transformer to obtain a key feature matrix and a value feature matrix representing text feature information; An initial label matrix acquisition module is used to construct a label tree based on the five-level hierarchical label system corresponding to the user opinion data, and input the label tree into a directed graph convolutional neural network to obtain an initial label matrix representing the hierarchical semantic structure information; The query feature matrix acquisition module is used to input the initial label matrix into the first multi-head attention module of the Transformer decoder to obtain the query feature matrix; The label matrix acquisition module after feature interaction is used to jointly input the key feature matrix, value feature matrix and query feature matrix into the second multi-head attention module of the Transformer decoder to obtain the label matrix after feature interaction; The user opinion stream data label category determination module is used to pass the label matrix through the fully connected layer to obtain the label category probability of the user opinion stream data, and take the label with a probability value greater than the set threshold as the hierarchical topic category of the user opinion stream data.

9. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium being used to store computer instructions, wherein when the computer instructions are executed by a processor, the method for hierarchical topic identification for user opinion stream data as described above is implemented; or, A computer program product includes a computer program, which is used to implement the above-mentioned hierarchical topic identification method for user opinion stream data when running on one or more processors.

10. The present invention provides an electronic device, comprising: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the hierarchical topic identification method for user opinion stream data as described above.

Images