A multi-value chain problem text classification method and device

Abstract

The application discloses a multi-value chain problem text classification method, comprising the following steps: collecting and marking the text category labels of the question sentences searched by users in each business search platform in an enterprise to obtain a data set of multi-value chain problem texts; preprocessing the data set and dividing the data set into a training text set and a test text set, and extracting word vectors; based on the training text set, constructing a full connection network graph by using the extracted word vectors and the marked text category labels, training an initial label graph attention model, and obtaining a trained label graph attention model; based on the word vectors extracted from the training text set, training an initial text semantic extraction model to obtain a trained text semantic extraction model; performing semantic extraction on the test text set according to the trained text semantic extraction model, and classifying the test text set according to the trained label graph attention model. The application can realize intelligent classification of multi-value chain problem texts.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text classification, in particular to a multi-value chain question text classification method and device. BACKGROUND

[0002] Under the continuous self-updating of data sharing platform in the multi-value chain in which enterprises work together, more and more enterprises have established their own knowledge bases to store important knowledge and experience in enterprise management, production, sales and other aspects, and intelligent question answering systems have been developed for enterprise workers to search the knowledge base, and enterprise workers tend to use natural language to query the desired knowledge, so classifying the questions is an important part of building a multi-value chain intelligent question answering system and plays an important role in promoting the intelligent question answering system service.

[0003] The core problem of text classification is text representation and classification model. The traditional text representation uses the Bag-of-Words principle to represent the words in a high-dimensional Vector Space Model without order, discarding text structure information such as word order and grammar. The traditional machine learning text classification model is based on the assumption that classification is subject to a certain probability distribution, and uses the Bayesian theory to obtain a classifier, but if the assumption does not hold, the classification accuracy will be affected. SUMMARY

[0004] The present application provides a multi-value chain question text classification method and device to realize intelligent classification of multi-value chain question texts.

[0005] The technical solution is as follows:

[0006] On the one hand, a multi-value chain question text classification method is provided, which comprises:

[0007] S1, collecting and labeling the text category tags of the questions searched by users in the business search platform in the enterprise to obtain a data set of multi-value chain question texts;

[0008] S2, preprocessing the question text data set, and dividing the preprocessed question text data set into a training text set and a test text set;

[0009] S3, extracting word vectors from the training text set and the test text set, and based on the training text set, constructing a full connection network graph using the extracted word vectors and the labeled text category tags;

[0010] S4, training an initial label graph attention model based on the full connection network graph to obtain a trained label graph attention model;

[0011] S5, training an initial text semantic extraction model based on the word vectors extracted from the training text set, to obtain a trained text semantic extraction model;

[0012] S6, performing semantic extraction on the test text set according to the trained text semantic extraction model, and classifying the test text set according to the trained label graph attention model.

[0013] Optionally, the preprocessing of the question text data set in S2 comprises:

[0014] Chinese word segmentation processing and stop word removal processing are performed on the question text data set;

[0015] The Chinese word segmentation processing and stop word filtering processing comprise:

[0016] The text is segmented based on a pre-constructed multi-value chain text expert dictionary, and a principle of maximizing the number of large granularity words is adopted to determine the segmentation result; wherein the text is a question text data set;

[0017] The stop words are filtered based on a pre-constructed multi-value chain stop word dictionary to remove noise words.

[0018] Optionally, the word vector extraction of the preprocessed training text set and test text set in S3, and the construction of a fully connected network graph based on the training text set and the extracted word vectors and labeled text category labels, comprise:

[0019] The word vectors of the preprocessed training text set and test text set are obtained based on a pre-trained BERT word vector extraction model;

[0020] Based on the training text set, the extracted word vectors and the labeled text category labels are used as nodes to construct a fully connected network graph between all nodes, wherein the nodes represented by the text category labels are initialized as random vectors with the same dimension as the word vectors, the edge weights between the word vector nodes are randomly initialized, and the edge weights between the category-word nodes are initialized as the probability of the occurrence of the word in the category in the training text set.

[0021] Optionally, the initial label graph attention model is trained to be a GNN-Attention model based on the fully connected network graph in S4, and the GNN-Attention model comprises a GNN sub-model and an Attention sub-model;

[0022] The GNN sub-model is used to obtain the output features of each category label node;

[0023] The Attention sub-model is used to control the attention weight distribution between nodes in the GNN sub-model.

[0024] Optionally, the training process of the GNN-Attention model comprises:

[0025] The feature vector of a node in the GNN-Attention model is iteratively updated according to the following formula (2):

[0026]

[0027] wherein σ0 represents a sigmoid activation function, W k represents a parameter matrix of the GNN at the kth iteration, k∈[1, K], wherein K∈R + , K>1, is a preset constant, N(v) represents all neighbor nodes of node v, |N(v)| represents the size of N(v), represents the feature vector of node u at the (k-1)th iteration, is the feature vector of node v at the (k-1)th iteration, is the output vector of node v at the kth iteration, B k is a bias parameter at the kth iteration, k is 1 representing the first iteration, and all parameters are randomly initialized. α vu represents the attention weight between node u and node v, and is defined as the following formula (3):

[0028]

[0029] wherein e vu is obtained according to the following formula (4):

[0030]

[0031] wherein a represents a function for calculating the attention weight;

[0032] The defined loss function is a class-balanced label similarity loss function, as shown in the following formula (5):

[0033]

[0034] wherein m represents the number of batches into which the entire training text set is divided, b represents the index of a batch, n b represents the number of training texts in the batch with index b, i represents the index of a training text in a batch, L represents the total number of class labels, s j represents the frequency of the jth label appearing in the entire training text set, and g b(i) represents the class label corresponding to the training text with subscript i in the batch with subscript b, z b (i) represents the node subscript corresponding to the training text with subscript i in the batch with subscript b in the full connection graph, z b (g b (i)) represents the class label g b (i) represents the node subscript corresponding to the training text in the full connection graph, W c represents the parameter matrix for linear transformation of the training text node, W l represents the parameter matrix for linear transformation of the label node, K is the total number of iterations mentioned above;

[0035] According to the above class-balanced label similarity loss function, the parameters in the GNN-Attention model are updated by using the batch gradient descent algorithm, and a trained label graph attention model is obtained.

[0036] Optionally, the word vector extracted based on the training text set in S5 is a Bi-LSTM model.

[0037] The Bi-LSTM sub-model is trained according to the following formula (6);

[0038]

[0039] Wherein, N represents the number of sentences in the training text set, y i represents the classification label vector of the i-th sentence, represented by one-hot encoding, p i represents the probability distribution of the final output of the i-th sentence, defined as the following formula (7);

[0040] p i =Softmax(z) (7)

[0041] Wherein, Softmax represents the Softmax activation function, z represents the dot product result vector of the output vector of the current text through the Bi-LSTM and each label vector, defined as the following formula (8);

[0042] z t =e i ·h t , t∈[0,M) (8)

[0043] Wherein z t represents the t-th component of the vector z, e i represents the output vector of the i-th text of the Bi-LSTM model, h t represents the representation vector of the t-th class label, and M represents the total number of class labels.

[0044] In another aspect, the present application provides a multi-value chain question text classification device, the device comprising:

[0045] A data collection module is configured to collect and label text category tags of question sentences searched by users in business search platforms in an enterprise to obtain a data set of multi-value chain question texts.

[0046] A data preprocessing module is configured to preprocess the question text data set and divide the preprocessed question text data set into a training text set and a test text set.

[0047] An extraction and construction module is configured to extract word vectors from the training text set and the test text set, and construct a full connection network graph based on the training text set, the extracted word vectors, and labeled text category tags.

[0048] A training module is configured to train an initial label graph attention model based on the full connection network graph to obtain a trained label graph attention model, and train an initial text semantic extraction model based on the word vectors extracted from the training text set to obtain a trained text semantic extraction model.

[0049] A classification module is configured to perform semantic extraction on the test text set according to the trained text semantic extraction model, and perform classification on the test text set according to the trained label graph attention model.

[0050] Optionally, the data preprocessing module is specifically configured to:

[0051] perform Chinese word segmentation processing and stop word removal processing on the question text data set.

[0052] The Chinese word segmentation processing and the stop word removal processing comprise:

[0053] segmenting the text based on a pre-constructed multi-value chain text expert dictionary, and determining a segmentation result by adopting a principle of maximizing the number of large granularity words; wherein the text is the question text data set.

[0054] filtering stop words based on a pre-constructed multi-value chain stop word dictionary to remove noise words.

[0055] Optionally, the extraction and construction module is specifically configured to:

[0056] obtain word vectors of the preprocessed training text set and the test text set based on a pre-trained BERT word vector extraction model.

[0057] Based on the training text set, the extracted word vectors and the labeled text category labels are used as nodes to construct a fully connected network graph between all nodes, wherein the node represented by the text category label is initialized as a random vector with the same dimension as the word vector, the edge weight between the word vector nodes is randomly initialized, and the edge weight between the category-word nodes is initialized as the probability of the occurrence of the word in the category in the training text set.

[0058] Optionally, the initial label graph attention model is a GNN-Attention model, and the GNN-Attention model comprises a GNN sub-model and an Attention sub-model.

[0059] The GNN sub-model is configured to obtain an output feature of each category label node.

[0060] The Attention sub-model is configured to control the attention weight distribution between nodes in the GNN sub-model.

[0061] Optionally, the training module is specifically configured to:

[0062] The feature vector of the node in the GNN-Attention model is updated iteratively according to formula (2) as follows:

[0063]

[0064] wherein σ0 represents a sigmoid activation function, W k represents a parameter matrix of the GNN in the kth iteration, k∈[1, K], wherein K∈R + , K>1, is a preset constant, N(v) represents all neighbor nodes of the node v, |N(v)| represents the size of N(v), represents the feature vector of the node u in the k-1th iteration, is the feature vector of the node v in the k-1th iteration, is the output vector of the node v in the kth iteration, B k is a bias parameter in the kth iteration, k is 1, indicating the first iteration, and all parameters are randomly initialized. vu represents the attention weight between the node u and the node v, and is defined as formula (3) as follows:

[0065]

[0066] wherein e vu is obtained according to formula (4) as follows:

[0067]

[0068] Wherein, a represents a function of calculating attention weight;

[0069] The defined loss function is a class balance label similarity loss function, as shown in the following formula (5):

[0070]

[0071] Wherein m represents the number of batches into which the entire training text set is divided, b represents the index of the batch, n b represents the number of training texts in the batch with index b, i represents the index of a training text in a batch, L represents the total number of class labels, s j represents the frequency of the jth label appearing in the entire training text set, g b (i) represents the class label corresponding to the training text with index i in the batch with index b, z b (i) represents the node index corresponding to the training text with index i in the batch with index b in the full connection graph, z b (g b (i)) represents the node index corresponding to the class label g b (i) in the full connection graph, W c represents the parameter matrix for linear transformation of the training text node, W l represents the parameter matrix for linear transformation of the label node, and K is the total number of iterations mentioned above.

[0072] According to the above class balance label similarity loss function, the batch gradient descent algorithm is used to update the parameters in the GNN-Attention model, and a trained label graph attention model is obtained.

[0073] Optionally, the initial text semantic extraction model is a Bi-LSTM model, and the training module is specifically configured to:

[0074] The Bi-LSTM submodel is trained according to the following formula (6):

[0075]

[0076] Wherein N represents the number of sentences in the training text set, y i represents the classification label vector of the ith sentence, represented by one-hot encoding, p i represents the probability distribution of the final output of the ith sentence, defined as shown in the following formula (7):

[0077] p i =Softmax(z) (7)

[0078] Wherein, Softmax represents a Softmax activation function, z represents a dot product result vector of an output vector of the current text through the Bi-LSTM and each label vector, and is defined as formula (8) as follows;

[0079] z t = e i ·h t , t e [0, M) (8)

[0080] Wherein, z t represents the tth component of the vector z, e i represents an output vector of the i th text by the Bi-LSTM model, h t represents a representation vector of the t th category label, and M represents the total number of category labels.

[0081] On the other hand, an electronic device is provided, which comprises a processor and a memory, the memory has at least one instruction stored therein, the at least one instruction is loaded and executed by the processor to realize the above-mentioned multi-value chain problem text classification method.

[0082] On the other hand, a computer readable storage medium is provided, the storage medium has at least one instruction stored therein, the at least one instruction is loaded and executed by a processor to realize the above-mentioned multi-value chain problem text classification method.

[0083] The technical scheme provided by the present application brings at least the following beneficial effects:

[0084] In the present application, firstly, the collection of multi-value chain problem text is carried out, which includes the construction of a multi-value chain field specific dictionary, and through the carding of specific terms in the multi-value chain field, great help can be provided for the accuracy of the segmentation of the text in the multi-value chain field.

[0085] Then, the problem text classification based on the graph attention network is carried out, the graph attention label model and the text semantic extraction model are trained to classify the search question of the enterprise staff in the multi-value chain field, which provides great help for the construction of the business question and answer system in the enterprise, the question and answer system can greatly reduce the search range and improve the search efficiency according to the problem classification result, and finally the question and answer system can obtain a faster response and more accurate semantic search. BRIEF DESCRIPTION OF DRAWINGS

[0086] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows, and obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0087] Figure 1 is a flow diagram of a multi-value chain problem text classification method provided by an embodiment of the present application;

[0088] Figure 2 is a detailed architecture diagram of a multi-value chain problem text classification model based on a graph attention network provided by an embodiment of the present application;

[0089] Figure 3 is a flow diagram of a multi-value chain problem text classification method provided by an embodiment of the present application;

[0090] Figure 4 is a block diagram of a multi-value chain problem text classification device provided by an embodiment of the present application;

[0091] Figure 5 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the technical problems, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the accompanying drawings.

[0093] The embodiment of the present application provides a multi-value chain problem text classification method, which can be realized by a multi-value chain problem text classification device. Figure 1 As shown in a flow diagram of a multi-value chain problem text classification method, the processing flow of the method can include the following steps:

[0094] S1, collecting and text category label marking of question sentences searched by users in each business search platform in an enterprise, to obtain a data set of multi-value chain problem texts;

[0095] S2, preprocessing the problem text data set, and dividing the preprocessed problem text data set into a training text set and a test text set;

[0096] S3, extracting word vectors from the training text set and the test text set, and based on the training text set, constructing a fully connected network graph using the extracted word vectors and the marked text category labels;

[0097] S4, training an initial label graph attention model based on the fully connected network graph, to obtain a trained label graph attention model;

[0098] S5, training an initial text semantic extraction model based on the word vectors extracted from the training text set, to obtain a trained text semantic extraction model;

[0099] S6, according to the text semantic extraction model trained, the test text set is carried out semantic extraction, according to the label graph attention model trained, the test text set is classified.

[0100] Optionally, the S2 in the pre-processing of the question text data set comprises:

[0101] The question text data set is subjected to Chinese word segmentation processing and stop word removal processing.

[0102] The Chinese word segmentation processing and stop word filtering processing comprise:

[0103] The text is segmented based on a pre-constructed multi-value chain text expert dictionary, and a principle of maximum number of large granularity words is used to determine the segmentation result; wherein the text is a question text data set.

[0104] The stop words are filtered based on a pre-constructed multi-value chain stop word dictionary, and noise words are removed.

[0105] Optionally, the S3 in the pre-processing of the training text set and the test text set comprises:

[0106] The pre-trained BERT word vector extraction model is used to obtain the word vector of the pre-processed training text set and the test text set.

[0107] Based on the training text set, the extracted word vector and the annotated text category label are used as nodes to construct a fully connected network graph between all nodes, wherein the node represented by the text category label is initialized as a random vector, the dimension is consistent with the dimension of the word vector, the edge weight between the word vector nodes is randomly initialized, and the edge weight between the category-word nodes is initialized as the probability of the occurrence of the word in the category in the training text set.

[0108] Optionally, the S4 in the pre-processing of the training text set and the test text set comprises:

[0109] The GNN sub-model is used to obtain the output feature of each category label node.

[0110] The Attention sub-model is used to control the attention weight distribution between nodes in the GNN sub-model.

[0111] Optionally, the training process of the GNN-Attention model comprises:

[0112] The feature vector of the node in the GNN-Attention model is updated iteratively according to the following formula (2);

[0113]

[0114] wherein σ0 represents a sigmoid activation function, W k represents a parameter matrix of the GNN at the kth iteration, k ∈ [1, K], wherein K ∈ R + , K > 1, is a preset constant, N(v) represents all neighbor nodes of the node v, |N(v)| represents the size of N(v), represents the feature vector of the node u at the k-1th iteration, is the feature vector of the node v at the k-1th iteration, is the output vector of the node v at the kth iteration, B k is a bias parameter at the kth iteration, k is 1 representing the first iteration, and all parameters are randomly initialized. α vu represents the attention weight between the node u and the node v, and is defined as the following formula (3);

[0115]

[0116] wherein e vu is obtained according to the following formula (4);

[0117]

[0118] wherein a represents a function for calculating the attention weight;

[0119] The defined loss function is a class balanced label similarity loss function, as shown in the following formula (5):

[0120]

[0121] wherein m represents the number of batches into which the entire training text set is divided, b represents the index of the batch, n b represents the number of training texts in the batch with the index b, i represents the index of the training text in a batch, L represents the total number of class labels, s j represents the frequency of the jth label appearing in the entire training text set, g b (i) represents the class label corresponding to the training text with the index i in the batch with the index b, z b (i) represents the node index corresponding to the training text with the index i in the batch with the index b in the full connection graph, z b (g b (i)) represents the class label gb (i) the node subscript corresponding in the full connection graph, W c represents the parameter matrix for linear transformation of the training text node, W l represents the parameter matrix for linear transformation of the label node, K is the total number of iterations mentioned above;

[0122] According to the above class balanced label similarity loss function, the batch gradient descent algorithm is used to update the parameters in the GNN-Attention model, and the trained label graph attention model is obtained.

[0123] Optionally, the word vector extracted based on the training text set in S5 is a Bi-LSTM model.

[0124] The Bi-LSTM submodel is trained according to the following formula (6);

[0125]

[0126] Where N represents the number of sentences in the training text set, y i represents the classification label vector of the i-th sentence, represented by one-hot encoding, p i represents the final output probability distribution of the i-th sentence, defined as the following formula (7);

[0127] p i =Softmax(z) (7)

[0128] Where Softmax represents the Softmax activation function, z represents the dot product result vector of the output vector of the current text through the Bi-LSTM and each label vector, defined as the following formula (8);

[0129] z t =e i ·h t , t∈[0,M) (8)

[0130] Where z t represents the t-th component of the vector z, e i represents the output vector of the i-th text of the Bi-LSTM model, h t represents the representation vector of the t-th category label, and M represents the total number of category labels.

[0131] In the embodiment of the application, first, the collection of multi-value chain problem texts is carried out, which includes constructing a multi-value chain field-specific dictionary, and through the carding of the multi-value chain field-specific terms, great help can be provided to the accuracy of the multi-value chain field text segmentation;

[0132] Then the problem text classification based on graph attention network is carried out, search questions of enterprise staff in the multi-value chain field are classified through training of a graph attention label model and a text semantic extraction model, and the construction of a business question and answer system in the enterprise is greatly helped. The question and answer system can greatly reduce the search range and improve the search efficiency according to the problem classification result, and finally the question and answer system can obtain a faster response and more accurate semantic search.

[0133] The embodiment of the application is described in detail below:

[0134] Firstly, the related knowledge in the field of text classification is introduced:

[0135] Text classification is an important part of text mining, and its main task is to divide a given text set into one or more known category sets. At present, the text classification task has been applied to many fields, such as sentiment analysis, question classification, spam detection and the like. The construction of a text classification system is the same as supervised learning, which needs a training set D=(d1, d2,..., d N ), wherein N is the number of sequences, each text is labeled with its category (c1, c2,..., c N ), and then a classification model is learned by using the training data, so as to predict the category of a new text x.

[0136] The core problem of text classification is text representation and classification model. The traditional text representation uses the bag-of-words principle (Bag-of-Words) to represent the words in a high-dimensional vector space model (Vector Space Model) without order, and discards the text structure information such as word order and grammar. The traditional machine learning text classification model is based on the assumption that classification is subject to a certain probability distribution, and uses the Bayesian theory to obtain the classifier, but if the assumption does not hold, the classification accuracy will be affected. With the development of deep learning, more and more neural network models are used for text classification. For example, fastText stacks and averages the word and n-gram vectors of the entire document to obtain a document vector, and then uses the document vector to do softmax multi-classification. However, the network result in fastText does not consider the word order information at all, and the n-gram feature used by it just shows the importance of local sequence information. Therefore, TextCNN is proposed, which applies convolutional neural network CNN to the text classification task, uses multiple kernels of different sizes to extract key information in the sentence (similar to n-gram of multiple window sizes), so as to better capture the local correlation.

[0137] Although TextCNN can perform well in many tasks, the biggest problem of CNN is that the view of fixed filter_size cannot model longer sequence information, and the hyperparameter adjustment of filter_size is also cumbersome. CNN is essentially a feature expression of text, while recurrent neural networks are more commonly used in natural language processing, which can better express context information.

[0138] Although CNN and RNN can be used in text classification tasks and have significant effects, they are not good at interpretability. The attention mechanism is a commonly used modeling long-term memory mechanism in the field of natural language processing, which can intuitively give the contribution of each word to the result. For example, the Hierarchical Attention Network divides a document into sentences and words in a hierarchical manner, and uses the attention mechanism in each layer, which has very good effect. These deep learning models can capture the semantic and syntactic information in local continuous word sequences well, but in the corpus with non-continuous and long-distance semantics, the co-occurrence of global words may be ignored, and the GNN model based on graph neural network can solve this problem well through iterative update. Therefore, the combination of GNN and Attention can solve the problems of feature expression and interpretability, and the variant Bi-LSTM of RNN is used for text semantic extraction.

[0139] The multi-value chain problem text classification scheme proposed in the embodiment of the application applies supervised learning and graph neural network learning theory, learns the features of the class labels by connecting the words in the training text and all nodes representing the class labels to form a full connection graph, then extracts the text features by training the text semantic extraction model, and finally obtains the label distribution of the text category by the similarity between the text features and the class label features, and takes the maximum probability as the predicted category, as shown in Figure 3 .

[0140] Based on the above idea, the flow chart of the multi-value chain problem text classification method based on graph attention network is as shown in Figure 2 , and the processing flow of the method can include the following steps:

[0141] S1, collect the question sentences searched by users in each business search platform in the enterprise and label the text categories, and obtain a data set of multi-value chain problem texts;

[0142] S2, pre-process the problem text data set, and divide the pre-processed problem text data set into a training text set and a test text set;

[0143] Optionally, the preprocessing of the question text dataset in the steps comprises Chinese word segmentation processing and stop word removal processing.

[0144] The Chinese word segmentation processing and stop word filtering processing comprises:

[0145] The text is segmented based on a pre-constructed multi-value chain text expert dictionary, and a principle of maximum number of large granularity words is adopted to determine the segmentation result; wherein the text is a question text dataset.

[0146] In a feasible implementation, since the multi-value chain field segmentation needs to be more accurate and accurate compared with the openness requirement of open field segmentation, the embodiment of the application adopts the expert dictionary construction segmentation method to segment the multi-value chain question text, and the dictionary is used to analyze the word level or short text field value in the document, extract the words or phrases, and combine the terms library accumulated by the enterprises in the multi-value chain and the general word library in the open field, to construct the professional segmentation word library in the field.

[0147] In a feasible implementation, by using the open source Jieba Chinese segmentation library, the personal dictionary is set as the content of the expert dictionary, and then the built-in segmentation algorithm is used, since the Jieba segmentation library provides a precise segmentation mode and a full mode segmentation, the latter displays all possible segmentation results, therefore, the embodiment of the application adopts the former, i.e., the precise segmentation mode, and the more accurate multi-value chain question text segmentation result can be obtained through segmentation.

[0148] The stop words are filtered based on a pre-constructed multi-value chain stop word dictionary to remove noise words.

[0149] In a feasible implementation, after text segmentation, the text has been segmented into word forms, but a large amount of low-quality data, called stop words, is contained, and these words have little influence on the text semantics, therefore, filtering the stop words is beneficial to reduce the data amount, and further reduce the calculation amount of subsequent work on text processing. The general stop word list includes numerals, quantifiers, conjunctions such as "of / ground / in / and", adverbs, etc., these words appear a lot in the segmentation result, and have little influence on the context semantics, therefore, they are removed.

[0150] The open field Chinese stop word list used in the embodiment of the present application includes: Harbin Institute of Technology stop word library, Baidu stop word library and Sichuan University Machine Intelligence Laboratory stop word list, and a multi-value chain field stop word list is constructed based on multi-value chain field expert knowledge. These word lists basically completely cover most of the Chinese stop words in the open field and the multi-value chain field stop words. By using the above complete common stop word library, most of the noise words irrelevant to the classification multi-value chain problem text can be effectively removed. After removing the stop words, the number of feature words of the problem text is greatly reduced, which can improve the model calculation speed in subsequent calculation and improve the accuracy of subsequent knowledge extraction based on the word vector model.

[0151] S3, performing word vector extraction on the training text set and the test text set, constructing a full connection network graph based on the training text set, the extracted word vector and the labeled text category label;

[0152] In a feasible implementation manner, after the multi-value chain problem text is segmented and the stop words are removed, a series of discrete Chinese words are obtained. These words cannot be processed by a computer. Therefore, a mathematical model is constructed to represent each word in the form of numbers that can be calculated by a computer, that is, natural language words are converted into digital vectors of a certain dimension to input the mathematical model for probability calculation, which is the process of extracting word vectors.

[0153] Optionally, the word vector extraction on the preprocessed training text set and the test text set in S3 can include the following steps:

[0154] Based on the pre-trained BERT word vector extraction model, the word vectors of the preprocessed training text set and the test text set are obtained, and the word vector extraction is completed.

[0155] Optionally, the training process of the BERT word vector extraction model includes the following steps S31-S33:

[0156] S31, constructing an initial BERT word vector extraction model and initializing its parameters, the input of the initial BERT word vector extraction model being a word sequence processed by a mask, and the word sequence processed by the mask being represented as where n is the sequence length, m1, m2,... m k are k subscripts randomly selected, k < n, and the words at these subscript positions are replaced by a mask word [MASK]; the output vector of the i-th position of the initial BERT word vector extraction model being the probability distribution p of predicting the output word of the corresponding position i ; wherein the word sequence is represented as , and the rest are initialized as random vectors, and the masked words are uniformly initialized as 0 vectors.

[0157] S32, define the loss function as a multi-class cross-entropy loss function, as shown in the following formula (1);

[0158]

[0159] S33, based on the multi-class cross-entropy loss function, the batch stochastic gradient descent method is used to update the parameters in the initial BERT word vector extraction model, and a pre-trained BERT word vector extraction model is obtained.

[0160] S4, based on the full connection network graph, the initial label graph attention model is trained to obtain a trained label graph attention model;

[0161] In an available embodiment, after the multi-value chain problem text is preprocessed by word segmentation, stop word removal and numerical conversion by word vectorization, a vector data table is obtained, the word vectors and category label vectors are used to form a full connection network graph, the dependency relationship between the words in the text and the category labels is modeled, and the feature vector representation of each category label is trained through graph convolution and attention mechanism. This is the technical focus of the multi-value chain problem text classification model in the embodiment.

[0162] Optionally, the initial label graph attention model is a GNN-Attention model, and the GNN-Attention model includes a GNN sub-model and an Attention sub-model.

[0163] The GNN sub-model is used to obtain the output features of each category label node.

[0164] The Attention sub-model is used to control the attention weight distribution between nodes in the GNN sub-model.

[0165] Optionally, the training process of the GNN-Attention model includes:

[0166] The feature vectors of the nodes in the GNN-Attention model are iteratively updated according to the following formula (2);

[0167]

[0168] wherein, σ0 represents a sigmoid activation function, W k represents the parameter matrix of the GNN in the kth iteration, k∈[1, K], wherein K∈R + , K>1, is a preset constant, N(v) represents all neighbor nodes of the node v, |N(v)| represents the size of N(v), represents the feature vector of the node u in the k-1th iteration, It is the feature vector of node v in the (k-1)th iteration. B is the output vector of node v at the k-th iteration. k This is the bias parameter in the k-th iteration, where k = 1 indicates the first iteration, and all parameters are randomly initialized. α vu The attention weight between node u and node v is defined as shown in the following formula (3);

[0169]

[0170] Among them, e vu The definition is obtained as shown in formula (4) below;

[0171]

[0172] Here, 'a' represents a function for calculating attention weights.

[0173] Optionally, in this embodiment of the invention, 'a' is an inner product calculation function, defined as a(x, y) = x·y. The coefficient matrix in the model... Where n is the word vector dimension The bias coefficient matrix represents the output of the current iteration. Features of node v in the previous layer Multiply by the bias coefficient matrix B k The input vector is then weighted, summed, and multiplied by the attention weights of all neighboring nodes of node v, and finally passed through an activation function σ0. The coefficient matrix W maps the input vector to the hidden layer, and its i-th column represents the transformation vector of the i-th term of the weighted sum of neighboring nodes. The activation function σ0 maps the hidden layer to the output layer, obtaining each term of the output vector.

[0174] The defined loss function is the class-balanced label similarity loss function, as shown in the following formula (5):

[0175]

[0176] Where m represents the number of batches into which the entire training text set is divided, b represents the batch index, and n b The index b represents the number of training texts in the batch, i represents the index of the training texts in a given batch, L represents the total number of category labels, and s represents the number of training texts in the batch. j G represents the frequency of the j-th label appearing in the entire training text set. b (i) represents the category label corresponding to the training text with index i in the batch with index b, z b (i) represents the node index in the fully connected graph corresponding to the training text with index i in the batch with index b, z b (g b(i) represents the category label g b (i) the node subscript corresponding in the full connection graph, W c represents the parameter matrix for linear transformation of the training text node, W l represents the parameter matrix for linear transformation of the label node, K is the total number of iterations.

[0177] According to the above category balanced label similarity loss function, the batch gradient descent algorithm is used to update the parameters in the GNN-Attention model, and a trained label graph attention model is obtained.

[0178] S5, based on the word vector extracted from the training text set, the initial text semantic extraction model is trained to obtain a trained text semantic extraction model;

[0179] In a feasible implementation, the text semantic extraction model training first annotates the multi-value chain problem text with problem text categories; then performs text segmentation, removes stop words, and extracts word vectors to obtain word vector representations of the text; according to the word vectors extracted from the training text, through a forward LSTM network and a backward LSTM network, the forward LSTM network extracts text features from left to right, and the backward LSTM network extracts text features from right to left, and then the two kinds of features are spliced to form the final text semantic features; the similarity between the text semantic features and the vector features of the category label in the text is used to obtain the predicted category, and the model is iteratively updated through the cross-entropy loss function and the gradient descent algorithm.

[0180] Optionally, the training process of the Bi-LSTM model includes:

[0181] The Bi-LSTM sub-model is trained according to the following formula (6);

[0182]

[0183] Wherein, N represents the number of sentences in the training text set, y i represents the classification label vector of the i-th sentence, represented by one-hot encoding, p i represents the probability distribution of the final output of the i-th sentence, defined as the following formula (7);

[0184] p i =Softmax(z) (7)

[0185] Wherein, Softmax represents the Softmax activation function, z represents the dot product result vector of the output vector of the current text through the Bi-LSTM and each label vector, defined as the following formula (8);

[0186] zt = e i · h t , t e [0, M) (8)

[0187] where z t denotes the t-th component of the vector z, e i denotes the output vector of the i-th text by the Bi-LSTM model, h t denotes the representation vector of the t-th class label, M denotes the total number of class labels, e i · h t denotes the result of the dot product of the i-th text and the class label t, which is used to calculate the similarity between the text and the class label, and the probability that the text belongs to a certain class label can be obtained by normalizing through the Softmax function. The Softmax function is defined as shown in the following formula (9);

[0188]

[0189] where z denotes the input vector, C denotes the length of the input vector, z c denotes the c-th component of the vector z, z i denotes the i-th component of the vector z, and the left side of the formula represents the value of the i-th component of the output of the vector z after the Softmax function. Softmax is a multi-class version of the Sigmoid function, which can well normalize the distribution of the vector to obtain an output between 0 and 1, and the sum is 1. It can be regarded as the probability of predicting the class distribution.

[0190] The class label prediction result is the maximum of all p i , that is, the prediction result is given as follows formula (10);

[0191] o = argmax i∈[0，M) (p i ) (10)

[0192] where i represents the class label with the serial number i, and o represents the result of judging which class label is finally, represented by a digital serial number.

[0193] S6, according to the trained text semantic extraction model, the semantic extraction of the test text set is carried out, and according to the trained label graph attention model, the test text set is classified.

[0194] In a feasible implementation, based on the word vector of the test text set that has been extracted, the semantic extraction of the text is carried out by Bi-LSTM to obtain a text vector; and the dot product calculation of the text vector and the trained label vector is carried out to obtain the predicted text class probability distribution.

[0195] In an implementable embodiment, the multi-value chain problem text classification method trained based on the graph attention mechanism model can be described as follows: first, the multi-value chain problem text is labeled with problem text categories; then, text segmentation, stop word removal, and word vector extraction are performed to obtain the word vector representation of the text; a fully connected network graph is constructed according to the word vectors extracted from the training text and the category label nodes; a graph attention label network is trained based on the fully connected network graph; for input test problem text, first, text semantic feature extraction is performed, then the text semantic features and each category label feature are respectively calculated for similarity, the probability distribution of the text prediction category is obtained, and the category with the maximum probability is taken as the final output.

[0196] The following describes an embodiment for a test text "Who is the business manager of the market development department". After segmentation and stop word processing, the text becomes "market development department business manager who". Four words are extracted by the BERT pre-training word vector extraction model, and the word vector extraction dimension is 30. The four word vectors are:

[0197] Market development department:

[0198]

[0199] Business:

[0200] Manager:

[0201]

[0202] Who:

[0203] The extracted semantic feature vector is obtained by inputting the text semantic extraction model, and the semantic feature vector dimension is 50. The extracted semantic feature vector is:

[0204]

[0205] According to the aforementioned label graph attention model trained based on the training text set, the category feature vectors of each category label have been obtained. In this paper, there are 5 category labels, represented by 0, 1, 2, 3, and 4 respectively. Their category feature vectors are obtained by averaging the aforementioned of the last iteration during training of the training text set, where I is the index of the category label, and K is the maximum number of iterations, i.e., the feature vector Wherein m is the batch quantity in the aforementioned training text set. The obtained five label feature vectors are as follows:

[0206] Label 0:

[0207] Label 1:

[0208]

[0209] Label 2:

[0210]

[0211] Label 3: Label 4:

[0212]

[0213] According to the similarity calculation function, the normalized similarity of the text feature vector and each category feature vector is calculated as the probability of being predicted as the category, and the similarity calculation formula is sim(x, y) = 10 4 *(x·y)*(||x||·||y||) -1 , and then the softmax normalization is performed, and the calculation result is as follows:

[0214] [8.6843e-01, 4.5041e-04, 2.1220e-04, 1.3028e-01, 6.2073e-04]

[0215] The category with the maximum probability, i.e. label 0, is 0.86843, which is taken as the output, and the test is completed.

[0216] Figure 4 It is a kind of multi-value chain problem text classification model construction device based on graph attention network according to the example embodiment. The device 400 comprises a data collection module 410, a data preprocessing module 420, an extraction construction module 430, a training module 440 and a classification module 450.

[0217] The data collection module 410 is used for collecting and text category label marking of the question sentences searched by users in each business search platform in an enterprise, to obtain a multi-value chain problem text data set.

[0218] The data preprocessing module 420 is configured to preprocess the question text dataset, and divide the preprocessed question text dataset into a training text set and a test text set.

[0219] The extraction and construction module 430 is configured to perform word vector extraction on the training text set and the test text set, and construct a full connection network graph based on the training text set and extracted word vectors and labeled text category labels.

[0220] The training module 440 is configured to train an initial label graph attention model based on the full connection network graph to obtain a trained label graph attention model, and train an initial text semantic extraction model based on the word vectors extracted from the training text set to obtain a trained text semantic extraction model.

[0221] The classification module 450 is configured to perform semantic extraction on the test text set according to the trained text semantic extraction model, and perform classification on the test text set according to the trained label graph attention model.

[0222] Optionally, the data preprocessing module is specifically configured to:

[0223] perform Chinese word segmentation processing and stop word removal processing on the question text dataset.

[0224] The Chinese word segmentation processing and the stop word removal processing include:

[0225] perform word segmentation on the text based on a pre-constructed multi-value chain text expert dictionary, and determine a word segmentation result by adopting a principle of maximizing the number of large granularity words; the text is the question text dataset.

[0226] filter stop words based on a pre-constructed multi-value chain stop word dictionary, and remove noise words.

[0227] Optionally, the extraction and construction module is specifically configured to:

[0228] obtain word vectors of the preprocessed training text set and the test text set based on a pre-trained BERT word vector extraction model.

[0229] construct a full connection network graph in which all nodes are fully connected based on the training text set, the extracted word vectors and the labeled text category labels, wherein a node represented by the text category label is initialized as a random vector with a dimension consistent with that of the word vector, an edge weight between word vector nodes is randomly initialized, and an edge weight between a category-word node is initialized as a probability of occurrence of the word in the category in the training text set.

[0230] Optionally, the initial tag graph attention model is a GNN-Attention model, and the GNN-Attention model comprises a GNN sub-model and an Attention sub-model.

[0231] The GNN sub-model is configured to obtain an output feature of each category label node.

[0232] The Attention sub-model is configured to control the attention weight distribution between nodes in the GNN sub-model.

[0233] Optionally, the training module is specifically configured to:

[0234] The feature vector of the node in the GNN-Attention model is iteratively updated according to the following formula (2):

[0235]

[0236] wherein σ0 represents a sigmoid activation function, W k represents a parameter matrix of the GNN in the kth iteration, k ∈ [1, K], wherein K ∈ R + , K > 1, is a preset constant, N(v) represents all neighbor nodes of the node v, |N(v)| represents the size of N(v), represents the feature vector of the node u in the k-1th iteration, is the feature vector of the node v in the k-1th iteration, is the output vector of the node v in the kth iteration, B k is a bias parameter in the kth iteration, k is 1, and all parameters are randomly initialized. vu represents the attention weight between the node u and the node v, and is defined as the following formula (3):

[0237]

[0238] wherein e vu is obtained according to the following formula (4):

[0239]

[0240] wherein a represents a function for calculating the attention weight;

[0241] The defined loss function is a class-balanced label similarity loss function, as shown in the following formula (5):

[0242]

[0243] wherein m represents the number of batches into which the entire training text set is divided, b represents the index of the batch, and nb represents the number of training texts in the batch with subscript b, i represents the subscript of a training text in a batch, L represents the total number of category labels, s j represents the frequency of the jth label appearing in the entire training text set, g b (i) represents the category label corresponding to the training text with subscript i in the batch with subscript b, z b (i) represents the node subscript corresponding to the training text with subscript i in the batch with subscript b in the full connection graph, z b (g b (i)) represents the category label g b (i) represents the node subscript corresponding to the training text in the full connection graph, W c represents the parameter matrix for linear transformation of the training text node, W l represents the parameter matrix for linear transformation of the label node, K is the total number of iterations mentioned above;

[0244] According to the above class balanced label similarity loss function, the parameters in the GNN-Attention model are updated by using the batch gradient descent algorithm, and a trained label graph attention model is obtained.

[0245] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the application. The electronic device 500 can have relatively large differences due to different configurations or performances, and can include one or more processors (central processing units, CPUs) 501 and one or more memories 502. The memory 502 stores at least one instruction, which is loaded and executed by the processor 501 to implement the steps of the above-mentioned multi-value chain problem text classification method.

[0246] In an exemplary embodiment, a computer-readable storage medium, such as a memory including instructions, is also provided. The above-mentioned instructions can be executed by a processor in a terminal to complete the above-mentioned multi-value chain problem text classification method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0247] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware, which can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0248] The above merely describes preferred embodiments of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-value chain problem text classification method characterized by, The method comprises: S1, collecting and text category label marking of the question sentences searched by users in each business search platform in an enterprise, to obtain a data set of multi-value chain problem texts; S2, preprocessing the problem text data set, and dividing the preprocessed problem text data set into a training text set and a test text set; S3, extracting word vectors from the training text set and the test text set, and based on the training text set, constructing a fully connected network graph using the extracted word vectors and the marked text category labels; S4, training an initial label graph attention model based on the fully connected network graph to obtain a trained label graph attention model; S5, training an initial text semantic extraction model based on the word vectors extracted from the training text set to obtain a trained text semantic extraction model; S6, performing semantic extraction on the test text set according to the trained text semantic extraction model, and classifying the test text set according to the trained label graph attention model; In S4, the initial label graph attention model based on the fully connected network graph is a GNN-Attention model, and the GNN-Attention model comprises a GNN sub-model and an Attention sub-model; The GNN sub-model is used to obtain the output features of each category label node; The Attention sub-model is used to control the attention weight distribution between nodes in the GNN sub-model; The training process of the GNN-Attention model comprises: According to the following formula (2), the feature vectors of the nodes in the GNN-Attention model are iteratively updated; where σ0represents a sigmoid activation function, W k represents the parameter matrix of the GNN at the kth iteration, k ∈ [1, K], where K ∈ R + , K > 1, is a preset constant, N(v) represents all neighbor nodes of node v, |N(v)| represents the size of N(v), represents the feature vector of node u at the k-1th iteration, is the feature vector of node v at the k-1th iteration, is the output vector of node v at the kth iteration, B k is a bias parameter at the kth iteration, k is 1 representing the first iteration, all parameters are randomly initialized, α vu represents the attention weight between node u and node v, and is defined as formula (3) as follows: where e vu is defined as follows in equation (4) Wherein, a represents a function for calculating attention weight; The defined loss function is a class balanced label similarity loss function, as shown in the following formula (5): where m denotes the number of batches into which the entire training corpus is divided, b denotes the index of a batch, n b denotes the number of training texts in the batch with index b, i denotes the index of a training text within a batch, L denotes the total number of class labels, s j denotes the frequency of the j-th label in the entire training corpus, g b (i) denotes the class label corresponding to the training text with index i in the batch with index b, z b (i) denotes the node index in the fully connected graph corresponding to the training text with index i in the batch with index b, z b (g b (i)) denotes the class label g b (i) in the fully connected graph, W c denotes the parameter matrix for the linear transformation of the training text nodes, W l denotes the parameter matrix for the linear transformation of the label nodes, K is the total number of iterations. According to the above class balanced label similarity loss function, the parameters in the GNN-Attention model are updated using the batch gradient descent algorithm to obtain the trained label graph attention model.

2. The method of claim 1, wherein, In S2, the preprocessing of the problem text data set comprises: Chinese word segmentation processing and stop word removal processing are performed on the problem text data set; The Chinese word segmentation processing and stop word filtering processing comprises: Based on a pre-constructed multi-value chain text expert dictionary, the text is segmented, and the principle of maximum number of large granularity words is used to determine the segmentation result; wherein the text is the problem text data set; Based on a pre-constructed multi-value chain stop word dictionary, the stop words are filtered to remove noise words.

3. The method of claim 1, wherein, In S3, the word vectors of the preprocessed training text set and test text set are extracted, and based on the training text set, a fully connected network graph is constructed using the extracted word vectors and the marked text category labels, comprising: Based on the pre-trained BERT word vector extraction model, the word vectors of the preprocessed training text set and test text set are obtained; Based on the training text set, the extracted word vector and the labeled text category label are taken as nodes to construct a fully connected network graph between all nodes, wherein the node represented by the text category label is initialized as a random vector with the same dimension as the word vector, and the edge weight between the word vector nodes is randomly initialized, and the edge weight between the category-word nodes is initialized as the probability of the occurrence of the word in the category in the training text set.

4. A multi-value chain problem text classification device characterized by comprising: The device comprises: A data collection module is configured to collect question sentences searched by users in each business search platform in an enterprise and label text category labels to obtain a data set of multi-value chain problem texts; A data preprocessing module is configured to preprocess the problem text data set and divide the preprocessed problem text data set into a training text set and a test text set; An extraction and construction module is configured to extract word vectors from the training text set and the test text set, and construct a fully connected network graph based on the training text set and the extracted word vectors and the labeled text category labels; A training module is configured to train an initial label graph attention model based on the fully connected network graph to obtain a trained label graph attention model, and train an initial text semantic extraction model based on the word vectors extracted from the training text set to obtain a trained text semantic extraction model; A classification module is configured to perform semantic extraction on the test text set based on the trained text semantic extraction model and perform classification on the test text set based on the trained label graph attention model; The initial label graph attention model is a GNN-Attention model, and the GNN-Attention model comprises a GNN sub-model and an Attention sub-model; The GNN sub-model is configured to obtain output features of each category label node; The Attention sub-model is configured to control the attention weight distribution between nodes in the GNN sub-model; The training module is specifically configured to: update the feature vectors of the nodes in the GNN-Attention model according to the following formula (2); wherein σ0represents a sigmoid activation function, w k represents a parameter matrix of the GNN at the kth iteration, k ∈ [1, K], wherein K ∈ R + , K > 1, is a preset constant, N(v) represents all neighbor nodes of the node v, |N(v)| represents the size of N(v), represents a feature vector of the node u at the k-1th iteration, is a feature vector of the node v at the k-1th iteration, is an output vector of the node v at the kth iteration, B k is a bias parameter at the kth iteration, k is 1 representing the first iteration, all parameters are randomly initialized, a vu represents an attention weight between the node u and the node v, and is defined as the following formula (3). where e vu is defined as follows in equation (4) wherein a represents a function for calculating attention weights; The defined loss function is a class-balanced label similarity loss function, as shown in the following formula (5): where m denotes the number of batches into which the entire training corpus is divided, b denotes the index of a batch, n b denotes the number of training texts in the batch with index b, i denotes the index of a training text within a batch, L denotes the total number of class labels, s j denotes the frequency of the j-th label in the entire training corpus, g b (i) denotes the class label corresponding to the training text with index i in the batch with index b, z b (i) denotes the node index in the fully connected graph corresponding to the training text with index i in the batch with index b, z b (g b (i)) denotes the class label g b (i) in the fully connected graph, W c denotes the parameter matrix for the linear transformation of the training text nodes, W l denotes the parameter matrix for the linear transformation of the label nodes, K is the total number of iterations mentioned above; According to the class-balanced label similarity loss function, the parameters in the GNN-Attention model are updated using a batch gradient descent algorithm to obtain the trained label graph attention model.

5. The apparatus of claim 4, wherein, The data preprocessing module is specifically configured to: perform Chinese word segmentation processing and stop word removal processing on the problem text data set; The Chinese word segmentation processing and stop word removal processing comprise: segmenting the text based on a pre-constructed multi-value chain text expert dictionary, and determining the segmentation result by maximizing the number of large-granularity words; wherein the text is the problem text data set; filtering stop words based on a pre-constructed multi-value chain stop word dictionary to remove noise words.

6. The apparatus of claim 4, wherein, The extraction and construction module is specifically configured to: Based on the pre-trained BERT word vector extraction model, the word vectors of the pre-processed training text set and the test text set are obtained; Based on the training text set, the extracted word vectors and the labeled text category labels are used as nodes to construct a fully connected network graph between all nodes, wherein the node represented by the text category label is initialized as a random vector with the same dimension as the word vector, the edge weight between the word vector nodes is randomly initialized, and the edge weight between the category-word nodes is initialized as the probability of the occurrence of the word in the category in the training text set.

Images