A text classification method based on event tags

Abstract

The present application discloses a text classification method based on event tags, which relates to the field of computer technology and includes: obtaining a text and performing preprocessing; using a pre-trained recurrent neural network XLNet to convert the preprocessed text into a sequence of word vectors; inputting the sequence of word vectors into a pre-trained Transformer encoder to output a text vector encoding representing the semantics of the text; constructing an event tag set representing text categories according to the text classification task; taking the tag set as input and using a pre-trained graph neural network to obtain the hierarchical relationship between the tags and output a tag vector representing the semantics of the tags; fusing the text vector encoding and the tag vector through a dynamic collaborative filtering attention mechanism to obtain a fused feature; using the fused feature as input to train a neural network model for text classification; and using the trained neural network model for text classification. Aiming at the low text classification accuracy in the prior art, the present application improves the text classification accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a text classification method based on event tags. Background Art

[0002] With the rapid development of the Internet, text data has grown explosively. How to automatically classify and process massive texts has become an important research topic. Currently, text classification technology has been widely applied in many fields such as search engines, news classification, spam filtering, etc. In response to the problem of low classification accuracy in text classification technology, many research works have proposed improvement methods.

[0003] Traditional text classification methods mainly rely on manual feature engineering, where manually constructed text features are input into the classification model. However, such methods require a large amount of professional background knowledge, and the classification performance is prone to reach a bottleneck. In recent years, deep learning technology has brought new ideas to the text classification task. Using neural networks can automatically learn the semantic features of texts and build an end-to-end text classification framework, avoiding complex feature engineering and having a stronger representation ability for the model. Currently, the mainstream methods in the field of text classification are end-to-end frameworks based on deep neural networks, such as CNN, RNN, attention mechanisms, etc., which can directly learn semantic features from texts and then input them into the classification model. However, these methods mainly focus on the semantics of the texts themselves and cannot well combine the prior knowledge of text categories, resulting in room for improvement in classification accuracy.

[0004] In related technologies, for example, Chinese Patent Document CN114896388A provides a hierarchical multi-label text classification method based on hybrid attention. This method uses pre-trained word vectors as word embeddings and uses Bi-GRU to perform preliminary feature extraction on the input word embeddings; uses a graph convolutional neural network to model the hierarchical label structure system and generate label representations containing label correlations; uses convolutional neural networks with different convolutional kernel sizes to further perform local feature extraction of different granularities on the output of Bi-GRU, respectively perform max pooling and then splice them into a text feature, and further perform feature extraction on this text feature using attention based on label representations; at the same time, uses self-attention mechanism to perform global feature extraction on the output of Bi-GRU, adaptively fuse the text feature based on label representations and the text feature based on self-attention mechanism to obtain a text representation based on hybrid attention, and finally extract information between labels through a relation network and obtain the final classification result through a multi-layer perceptron. However, GCN depends on a well-defined hierarchical label structure. If the hierarchical relationship is defined inaccurately or the label graph structure does not fully represent the relationship between labels, this will affect the quality of label representations. Therefore, the text classification accuracy of this solution needs to be further improved. Summary of the Invention

[0005] Aiming at the problem of low text classification accuracy in the existing technology, the present application provides a text classification method based on event tags, which improves the text classification accuracy by comprehensively considering the semantics of the text and the tags.

[0006] The purpose of the present application is achieved through the following technical solutions.

[0007] An embodiment of this specification provides a text classification method based on event tags, including: obtaining a text, and performing preprocessing of word segmentation and stop word removal on the obtained text; using the pre-trained recurrent neural network XLNet to convert the preprocessed text into a sequence of word vectors; inputting the sequence of word vectors into the pre-trained Transformer encoder to output a text vector encoding representing the semantics of the text; constructing an event tag set representing the text categories according to the text classification task; using the tag set as input, and using the pre-trained graph neural network to obtain the hierarchical relationship between the tags and output a tag vector representing the semantics of the tags; fusing the text vector encoding and the tag vector through a dynamic collaborative filtering attention mechanism to obtain a fusion feature; using the fusion feature as input to train a neural network model for text classification; and using the trained neural network model for text classification to perform text classification.

[0008] Among them, word segmentation refers to the process of splitting a sentence or text into words or phrases with semantic boundaries. In the present application, word segmentation is performed on the obtained text to divide the text into individual words, which is one of the basic preprocessing steps for natural language processing tasks such as text classification. Word segmentation can remove punctuation marks in the sentence and split words or phrases in the text to facilitate subsequent operations such as establishing word vectors to represent the semantics of the text.

[0009] Among them, stop word removal refers to the process of filtering and removing some commonly used words with little semantic contribution in text preprocessing, and these words are called stop words. Stop words mainly include common words such as prepositions, auxiliary words, pronouns, and numeral-classifiers that are not very useful in text classification tasks. In the present application, stop word removal is performed on the word-segmented text to filter out some meaningless commonly used words, reduce the dimension of the subsequent feature vectors, and improve the efficiency of model training and prediction. Word segmentation and stop word removal are standard preprocessing steps in NLP tasks such as text classification, which can split phrases and filter out useless words to lay a foundation for subsequent operations representing the semantics of the text.

[0010] Among them, XLNet is a pre-trained language representation model based on Transformer. Its structure forms an autonomous regression loop that can model the dependencies between words and is pre-trained through two prediction tasks. The word vector sequence refers to mapping each word in the text into a dense vector representation of a fixed dimension and connecting them into a sequence according to the order in which the words appear in the text. In this application, the pre-trained XLNet model is used to learn the semantic feature representation of the text, convert the text into semantic vectors, and provide input for subsequent text classification. The pre-processed text is converted into a word vector sequence by XLNet, and each word is represented by a vector of a fixed length. A text forms a word vector sequence, which can facilitate the model to learn the semantic information of the text. Using the pre-trained XLNet to extract the semantic features of the text and converting the text into a word vector sequence as the model input is a common method for representing text in text classification tasks and can improve the model's ability to understand semantics.

[0011] Among them, Transformer is a neural network structure based on the attention mechanism. Its encoder part learns and extracts semantic information through multi-head attention and fully connected networks. Text vector encoding refers to using a neural network to map a text into a dense vector representation of a fixed length to represent the semantic information of the text. In this application, the pre-trained Transformer encoder model is used. The word vector sequence of the input text is input, and the encoder learns the context semantic information of the text and outputs the text semantic vector. The Transformer encoder learns the word vector sequence of the input text and outputs the text vector encoding representing the semantic information of the whole text, providing a semantic feature representation for subsequent text classification tasks. Using the pre-trained Transformer encoder can effectively learn the semantic information of the text, map the text into a semantic vector encoding of a fixed length, and provide semantic input for text classification tasks.

[0012] Among them, the event label set refers to constructing a set of different classification labels according to the text classification task, and each label represents the semantic category of a class of texts. In this application, for the text classification task, an event label set containing different classification labels is constructed, such as categories classified as "sports", "entertainment", "finance", etc. Each label in the label set represents a semantic category in the text classification task. According to the text classification task, such as classifying different news topics, the categories of text classification are determined. A corresponding semantic label is set for each classification category, such as "sports", "entertainment", etc. These labels representing different classification categories are aggregated to construct an event label set. This event label set provides input for subsequent representation of label semantics. The event label set is constructed according to the text classification task, contains semantic labels representing different classification categories, and provides category input for subsequent label semantic learning and text classification.

[0013] Among them, the graph neural network is a type of neural network that can model graph-structured data, and it uses the graph connection relationships between nodes for feature extraction and learning. The hierarchical relationship refers to the superior-subordinate, peer-level relationships, etc. existing between labels. The label semantics refers to the semantic information represented by the label or the semantic category of the label. In this application, a pre-trained graph neural network model is used, and the constructed event label set is input to learn the semantic relationships between labels. The graph neural network can learn the hierarchical relationships between the labels in the label set, such as the superior-subordinate relationship between "sports" and "football". The graph neural network learns the semantic information of each label node and outputs a label semantic vector representing the meaning of each label. By using the pre-trained graph neural network model to input the constructed event label set, the hierarchical relationships between labels, that is, the semantic associations between labels, are learned, and a label vector representing the semantic information of each label is output, providing semantic information for subsequent text classification.

[0014] Among them, the attention mechanism of dynamic collaborative filtering is an attention method that enables text features and label features to collaborate. It dynamically generates semantic fusion features through the collaborative filtering of the GetText attention score and the label attention score. In this application, the multi-head attention scores of the text are calculated to represent the importance of different parts of the text. The multi-head attention scores of the labels are calculated to represent the importance of different labels. The text attention score and the label attention score are multiplied element by element to achieve the dynamic fusion filtering of the two. The filtered attention scores are used to weight the text and label features to generate dynamic semantic fusion features. Such an attention mechanism can dynamically learn the correlation degree between the text and the label, perform collaborative semantic fusion, and obtain better feature expressions. The attention mechanism of dynamic collaborative filtering can effectively utilize the relevance between the text and the label, achieve dynamic fusion, and obtain richer semantic features.

[0015] Specifically, perform multi-head self-attention on the text to obtain the multi-head self-attention matrix A of the text; perform multi-head self-attention on the label to obtain the multi-head self-attention matrix B of the label; perform element-wise multiplication of the attention matrix A of the text and the attention matrix B of the label, i.e., A⊙B, to obtain the text-label interaction attention matrix; use this text-label interaction attention matrix as a filtering mechanism to perform weighted summation on the vectors of the text and the label to obtain the dynamic fusion representation of the text and the label. Among them, the element-wise matrix multiplication realizes the mutual filtering of the attention scores of the two vectors of the text and the label at different positions, thereby realizing the weighted fusion of dynamic collaboration. This filtering mechanism based on the interaction of multi-head self-attention matrices can dynamically learn the correlation degree between the text and each label at different positions and realize the dynamic collaborative fusion of the semantics of the text and the label. In comparison, the traditional splicing method does not consider the correlation degree between the text and the label at different positions, and the fusion effect is relatively rough.

[0016] Further, the pre-trained recurrent neural network XLNet includes: constructing a text sample {T}; constructing a recurrent neural network XLNet containing multiple layers of bidirectional Transformer encoders for learning the semantic features of the text; inputting the text sample {T} and using a factorization algorithm to factorize the text sample {T} to generate m word order combinations {w i} of the text sample {T}; performing permutation and combination on the generated m word order combinations {w i} to obtain m semantic combinations {s i}; setting the training objective of the recurrent neural network XLNet, where the training objective includes a sentence order prediction task T1 and a word prediction task T2; training the recurrent neural network XLNet using the text sample {T} and the training objective.

[0017] Specifically, a large-scale text corpus is collected as training data, and the text is preprocessed, including word segmentation, building a dictionary, etc. The preprocessed text is constructed into a text sequence {T}, where T represents the preprocessed text dataset. A loop neural network XLNet is constructed that includes multiple layers of bidirectional Transformer encoders: XLNet is composed of multiple layers of bidirectional Transformer encoders stacked together. Each layer of the Transformer encoder contains a multi-head attention module and a feed-forward fully connected network. The multi-head attention module allows the model to learn the dependencies between words, and the feed-forward fully connected network performs non-linear mapping. By constructing a bidirectional context, the Transformer encoder can capture the bidirectional dependencies of words. Stacking multiple layers of Transformer encoders forms a deep network, which improves the learning ability of the model. XLNet implements a predictive autoregressive sequential prediction mechanism, forming a loop structure. This structure allows XLNet to learn word order information and model context dependencies. XLNet learns text semantic information through a context encoding module.

[0018] Among them, the factorization algorithm is a method of decomposing the input into multiple factors, and the product of these factors can approximate or be equal to the original input. In this application, the input text sample T is factorized into multiple word order combinations {wi}. Each word order combination contains all the vocabulary of the input text T, but the word order is different. Multiple combinations of word orders can be generated through factorization. The factorization algorithm can generate word order combinations using probability modeling. The factorization algorithm can effectively split the input text and generate multiple word order combinations with different word orders but the same vocabulary, which provides diverse semantic inputs for the pre-training of XLNet.

[0019] Specifically, for the input text sample T, a factorization method of probability modeling is used to factorize it into m sequences of word order combinations {w i}. Each word order combination w i contains all the vocabulary of the input text T, but the word order is randomly arranged and combined. The factorization algorithm randomly generates word order combinations by setting a permutation probability distribution: Set the set of all permutations of the input vocabulary as S, define the probability of each permutation as p(s), s ∈ S, and randomly sample according to the probability distribution p(s) to generate word order combinations. After generating m word order combinations {w i}, they are input into XLNet for the sequential prediction task T1. The T1 task trains XLNet to predict the original order according to the word order combination w i and capture bidirectional context information. The m word order combinations provide diverse inputs, improving the robustness of XLNet in learning semantic representations. Finally, through a large amount of training, XLNet obtains a more perfect semantic feature extraction ability.

[0020] Specifically, the generated m word order combinations are permuted to obtain semantic combinations, and the input text T is factorized to generate m word order combinations {w1, w2,......, w m}. Each word order combination contains all the vocabulary of the input text, but the order is randomly arranged. By permuting these m word order combinations, more semantic combinations can be obtained. Specific permutation method: randomly select k word order combinations, perform permutation operations to obtain new word orders, and use the new word orders as a semantic combination, denoted as s i , repeating this process, multiple semantic combinations {s1, s2,......, s k} can be obtained. The number of permutations is Cmn, that is, selecting n from m word order combinations for permutation can generate enough semantic combinations. These semantic combinations are used as new training samples and input into XLNet. XLNet learns these semantic combinations to obtain a richer semantic feature representation. Ultimately, it improves XLNet's ability to model semantic information and enhances the performance of downstream tasks.

[0021] Specifically, the training objective of XLNet includes two tasks: the sentence order prediction task T1 and the word prediction task T2. Sentence order prediction task T1: Input a sentence with scrambled word order, and the model predicts the original sentence order to train the model to learn bidirectional context dependencies. Word prediction task T2: Mask some words in the input sentence, and the model predicts the masked words to train the model to learn semantic representations. XLNet joint training: Input the processed text samples {T} into both tasks simultaneously, update the model parameters through the backpropagation algorithm, and alternately train the two tasks to jointly optimize XLNet's semantic extraction ability. After repeating multiple rounds of training, XLNet learns to model the text semantics and order and can encode semantic information. During fine-tuning, directly load the XLNet parameters and apply them to downstream tasks to improve the effect.

[0022] Furthermore, the sentence order prediction task T1 includes: obtaining m word order combinations {w i}; randomly selecting n word positions in the word order combination {w i} and swapping the order of the words, repeating the swap A times to obtain the word order combination {w' i}; according to the word order combination {w' i}, using the recurrent neural network XLNet to output the predicted word order {p(w i )}; calculating the loss loss1 between the word order {p(w i )} and the word order combination {w i} as the error; according to the loss loss, updating the parameters of the recurrent neural network XLNet through the gradient descent algorithm.

[0023] Specifically, randomly select the positions of the words in the word order combination for swapping. Input the word order combination {w i}, which contains all the words of the text sample, but the order is randomly arranged. Set the hyperparameter n, which represents the number of words swapped each time. Repeat the following process A times: Randomly select the positions of n words in the word order combination {w i}. Swap the words at these n positions to generate a new word order combination {w' i}. Word swapping method: Randomly select positions {p1, p2,......, p n}, save the words at these n positions as {tmp1, tmp2,......, tmp n}, and replace the word at the i-th position with tmp_(i + 1) % n. Repeat the above process A times to obtain multiple word order combinations {w' i}. Use these word order combinations {w' i} as training data and input them into XLNet. XLNet learns these word order combinations to obtain robustness to word order scrambling. Finally, improve the effect of XLNet in processing natural language.

[0024] Specifically, use XLNet to predict the order of the word order combination. Among them, in this application, the input is: the word order combination {w' i} with scrambled word order. The XLNet network structure: includes multiple layers of bidirectional Transformer encoders, and realizes a cyclic structure through autoregression and predicting the order. The encoder contains a self-attention mechanism to learn semantics. Order prediction process: Input the word order combination {w' i} into XLNet. Each layer of the encoder outputs the representation of the word at this position. Project the output at each time step through a linear layer to calculate the probability {p(w i )} of each word order, and output the most likely predicted word order {p(w i )}. Model training: Calculate the loss between the predicted word order {p(w i )} and the original word order w i , update the parameters through backpropagation, repeat multiple rounds of training to improve the prediction ability. Through the order prediction task, XLNet learns to encode context semantic information, obtains semantic feature expressions, and improves the effect of downstream tasks. Specifically, calculate the loss between the predicted word order and the original word order as the error. Input: Predicted word order: {p(w1), p(w2),......, p(w n )}, original word order: w1, w2,......, w n . Loss calculation: Traverse each word position i, and calculate the predicted word order {p(w i )} and the original word order w iThe cross-entropy loss between them. Cross-entropy loss function: loss i = -log(p(w i ))), sum up the losses at each position to get the total loss: loss1 = Σloss i , Optimization process: Calculate the gradient of loss1 with respect to the model parameters through the backpropagation algorithm, use an optimization algorithm (such as Adam) to update the parameters, minimize loss1, repeat multiple rounds of training to improve the prediction accuracy, and loss1 decreases. Finally, the training of the word order prediction task enables XLNet to learn semantic features. loss1 measures the model prediction error and guides the training optimization.

[0025] Specifically, the loss function: loss is used to measure the training error of XLNet, such as the cross-entropy loss loss1 of the word order prediction task. Parameter update: Calculate the gradient of loss with respect to the XLNet model parameters Θ represents all the learnable parameters in XLNet, calculated through the backpropagation algorithm. Gradient descent: Update the model parameters Θ using the gradient grad: Θ' = Θ - lr × grad, where lr is the learning rate that controls the update step size, and the gradient direction indicates the direction where the loss decreases fastest. Repeat training: Repeat the above parameter update for each batch of training data. After traversing all the training data, complete one training epoch. Cycle through multiple epochs to gradually reduce loss and complete the model training. Finally, through the gradient descent algorithm, the XLNet model parameters are iteratively optimized to learn the modeling of semantic sequences and complete the pre-training.

[0026] Furthermore, calculate the loss loss1, loss1 = -Σ(λ × log(p(w i )) + (1 - λ)D), where {p(w i )} is the predicted probability of the original word order w i ; λ is a coefficient with a value range of 0 to 1; D is a constant term. Specifically, λ is used to balance the two losses, and the value range of λ from 0 to 1 can adjust the ratio. The first term is the original cross-entropy loss, and the second term is the constant term. When λ = 1, it degenerates to the original loss1. When λ decreases, the weight of the first term loss1 decreases. The constant term plays a certain regularization role. By adjusting λ, the loss function can be balanced to a certain extent: against overfitting, prevent directly memorizing the input, and smooth the optimization difficulty of the loss function. Selecting an appropriate λ can help XLNet obtain more stable and effective semantic feature extraction ability during pre-training.

[0027] Furthermore, the word prediction task T2 includes: obtaining m word order combinations {w i}; Construct a dual-channel neural network, including a first channel and a second channel; Randomly select B words from the word order combination {w i} for masking processing to generate a masked sample {T'}; Use the masked sample {T'} as the input and input it into the first channel to output the prediction result {p i} of the masked word; Randomly select C words from the word order combination {w i} for tagging processing to obtain the tagged words {w m}; Use the word order combination {w i} and the tagged words {w m} as the input and input them into the second channel to output the context semantic representation {c m} corresponding to the tagged words {w i} in the word order combination {w i}.

[0028] Calculate the prediction loss loss2 of the word: loss2 = loss 21 (p i ) + λloss 22 (c i ), where loss 21 is the prediction loss of the first channel, and loss 22 is the context loss of the second channel; Update the parameters of the recurrent neural network XLNet according to loss2 through the gradient descent algorithm.

[0029] Among them, the dual-channel neural network refers to a neural network structure including two relatively independent sub-network channels. The two sub-network channels process different input information respectively, and finally the channel outputs are fused to obtain the final output result. In this application, a neural network with two channels is constructed. The first channel processes the data of the text, and the second channel processes the data of the label. The two channels learn semantic feature representations respectively, and finally the feature representations of the two channels are fused for the final text classification. Construct the first channel network to learn the semantic feature representation of the text data. Construct the second channel network to learn the semantic feature representation of the label data. The two-channel network structure can adopt a convolutional neural network, etc. The last layer fuses the feature representations of the two channels for the text classification task. The dual-channel structure can focus on learning the features of two different types of data respectively. Constructing a dual-channel neural network can learn the semantic features of text data and label data simultaneously, and fuse the information of both to improve the performance of text classification.

[0030] Among them, the masking process refers to randomly selecting some words in the input text, masking the content of these words, and usually replacing these words with specific markers (such as [MASK]). A masked sample refers to an example generated by performing a masking process on the original input sample. Some content in the masked sample is replaced by the masking marker. In this application, B positions of words are randomly selected from the word order combination {w i}, and the words at these positions are replaced with [MASK]. The word order combination {w i} that has undergone the masking process is used as the masked sample {T'}. {T'} is the word order combination with B words masked. By randomly masking words in the word order combination, a masked sample can be obtained, which can be used to train a neural network to predict the original word at the masked position, improve the semantic modeling ability of the model, and provide better semantic features for downstream tasks.

[0031] Specifically, the masked sample is input into the first channel for masked word prediction. Generation of the masked sample {T'}: Randomly mask B words in the word order combination {w i}, and use the [MASK] marker to mark the masked words to obtain the masked sample {T'}. Structure of the first channel: It includes modules such as a word vector layer and a BiLSTM layer, and can load pre-trained language models such as BERT / XLNet to learn text semantic feature representations. Masked word prediction process: Input {T'} into the first channel, and the channel outputs the word vector representation of each position. For the masked word, the predicted word vector {p i} is output, and {p i} represents the prediction result of the masked word. Model training: Calculate the loss between {p i} and the original word vector, perform backpropagation training, and the parameters of the channel are updated to gradually improve the prediction accuracy. Finally, the masked word prediction task enables the first channel to obtain text semantic feature expressions and provides inputs for subsequent text matching.

[0032] Among them, the marking process refers to randomly selecting some words in the input text and performing annotation or marking in a specific way to distinguish these words. Marked words refer to the words marked in the input text through the marking process. Marked words usually have specific marking methods. In this application, C words are randomly selected from the word order combination {w i}, and the special marker Obtainingmaskedwords{w m} is added as the mark for these words. {w m} represents the C words with marks selected from {w i}. Through the marking process, some words can be selected from the word order combination and marked to distinguish these words. Marked words can be used to train the model to learn to pay attention to specific words, thereby enhancing the semantic modeling ability of the model.

[0033] Specifically, the word order combination and the marked words are input into the second channel to obtain the context representation of the marked words. The word order combination is {w i}, and the marked words are {w m}, which are marked from {w i}. The structure of the second channel: It includes modules such as BiLSTM and attention mechanism to learn the context semantics of the marked words. The process: {w i} and {w m} enter the second channel as input sequences. BiLSTM learns the context representation of each word. For the words in {w m}, the attention mechanism is used to focus on their context to obtain the context semantic representation {c i} of the marked words. Model training: Define the loss function to make {c i} semantically consistent with {w m}, and backpropagate to train the parameters of the second channel, gradually improving the semantic focusing ability of the context representation of the marked words. Finally, the second channel can output the context focused semantic representation {c i} of the marked words, providing input for subsequent text matching. The second channel specifically learns the context semantics of the marked words and outputs the semantic representation {c i} of the marked words.

[0034] Specifically, calculate the word prediction loss and update the XLNet parameters. Calculate the prediction loss loss2 of the words: loss2 = loss 21 (p i ) + λloss 22 (c i ), where loss 21 is the prediction loss of the first channel, and loss 22 is the context loss of the second channel; update the parameters of the recurrent neural network XLNet according to loss2 through the gradient descent algorithm. Parameter update: Calculate the gradient of loss2 with respect to the XLNet parameters: Θ represents all the learnable parameters of XLNet, and calculate the gradient grad through the backpropagation algorithm. Gradient descent: Use the gradient grad to update the parameters: Θ' = Θ - lr × grad, where lr is the learning rate, controlling the update step size. Repeated training: Each time a batch is passed in, perform the above update process. After traversing all the training data, complete one training epoch. Repeat multiple epochs to optimize the parameters Θ of XLNet. Finally, under the joint optimization of the two losses, the XLNet parameters Θ learn the semantic feature representation, providing an optimized pre-trained language model for downstream tasks. In summary, defining the composite loss and applying gradient descent can effectively optimize the training of XLNet and obtain the ability to model text semantics.

[0035] Furthermore, loss 21 is the prediction loss of the first channel:

[0036] loss 21 (p i ) = -Σ(y × log(p i ) + (1 - y) × log(1 - p i )) + αR(θ), where y is the true one-hot label of B masked words; p i is the prediction probability of the first-channel model for these B words; θ represents the parameters, R(θ) represents the L2 regularization term of the parameters θ; α represents the regularization term weight hyperparameter;

[0037] loss 22 is the context loss of the second channel: where c i represents the context representations of C token words output by the second channel; z i represents the true context representation corresponding to the token words; d(c i , z i ) represents the distance function between c i and z i ; sim(c i , z i ) represents the similarity function of c i and z i ; γ, β are loss term weight hyperparameters.

[0038] Specifically, for the calculation of the masked word prediction loss of the first channel, it is defined that: y represents the one-hot label of the masked word, p i represents the prediction probability of the first channel for the masked word, α is the regularization weight hyperparameter, and R(θ) represents the L2 regularization term. Cross-entropy loss: Calculate the cross-entropy between the masked word prediction probability p i and the label y, and the formula is:

[0039] -Σ(y × log(p i ) + (1 - y) × log(1 - p i )) represents the measure of the error between the prediction p i and the true label y. Add L2 regularization: Calculate the L2 regularization term R(θ) of the parameters θ to avoid overfitting, and the weight is the hyperparameter α. Combine the two parts: loss 21 = cross-entropy loss + αR(θ), which considers both the prediction error and the model complexity. By calculating loss 21 and backpropagating, the masked word prediction model of the first channel can be trained to obtain the representation of the text semantics.

[0040] The context loss refers to the training loss of calculating a model's context representation of a specific word (usually a labeled word) in a text. It supervises the model to learn the context of words by defining the distance or similarity between the context representation and the true context representation. In this application, loss 22 It is the training loss of the context representation of the second channel tag word. It calculates the context representation of the tag word c output by the second channel i The corresponding true context representation z i The distance between them, and the similarity penalty term between them. This loss function is used to optimize the second channel model to obtain the modeling of the context of the labeled words. λ is used to balance the first channel prediction loss loss 21 and the second channel context loss loss 22 , loss 21 Used to train the first channel to predict masked words, loss 22 It is used to train the second channel to learn the context of labeled words. The two are optimized together to obtain a complete modeling of the text semantics.

[0041] Specifically, the calculation of the context loss of the second channel tag word is defined as: c i represents the context representation of the token word output from the second channel, z i represents the true context representation corresponding to the marked word, d(c i ,z i ) means c i and z i The distance function between i ,z i ) means c i and z i The similarity function between them. γ and β are the weight hyperparameters of the loss term. Distance loss: Calculate c i and z i The distance d(c i ,z i ) represents the difference in context representation, and the distance loss is obtained by averaging all the labeled words. Similarity penalty: Calculate c i and z i The similarity between sim(c i ,z i ) indicates that the similarity is too high to avoid the destruction of the objective function. L2 regularization: Add L2 regularization with parameter θ to penalize the model complexity, combining three parts: loss 22 = distance loss - similarity penalty + L2 regularization, training the context representation of the second channel tag words, by calculating loss 22 By back-propagating, the second channel model can learn the contextual semantic representation of the labeled words.

[0042] Further, taking the label set as the input, using a pre-trained graph neural network to obtain the hierarchical relationship between labels, and outputting label vectors representing label semantics, including: using the event labels in the text classification task as nodes and the semantic relationships between event labels as edges to construct a label graph; according to the label graph, performing node classification and negative sampling through the graph neural network to obtain the hierarchical relationship between event labels; for each label in the event label set, in the graph neural network, taking the label as the central node and aggregating neighboring nodes having a hierarchical relationship with the central node as the label vector of the corresponding node.

[0043] The graph convolutional network includes: an input layer that receives the event label set as input, a graph convolutional layer that learns the semantic relationships between the event label sets; a pooling layer that reduces the dimensionality of node features through the maximum pooling algorithm; a fully connected layer that maps the reduced-dimensional node features to label vectors representing the semantics of event labels; and an output layer that outputs label vectors representing label semantics.

[0044] Among them, the label graph is a graph structure, where nodes represent classification labels and edges represent the semantic relationships between different labels. The label graph can visually display the associations and semantic information between labels. In this application, using the event labels in the text classification task as nodes and the semantic correlation degree between two nodes as edge weights to construct a graph structure of the relationships between event labels, and obtaining the label graph. The nodes are the event category labels of the text classification task, and the edges represent the semantic correlations between different event labels. Constructing the label graph can visually reflect the label semantic information. The label graph can be used to provide label context information and enhance the text classification performance. Constructing the label graph can effectively represent the label semantic information in the event classification task and provide additional label context knowledge for subsequent text classification.

[0045] Among them, node classification is to classify the nodes in the graph in the graph neural network to distinguish different types of nodes. Negative sampling is to sample negative class samples for nodes other than positive samples during the training process to optimize model training. In this application, in the label graph, classifying the event label nodes and distinguishing nodes at different levels according to the edge relationships. At the same time, performing negative sampling, constructing negative sample edges for non-adjacent node pairs, and training the graph neural network model. Classifying the nodes in the label graph into different levels, performing negative sampling on non-adjacent node pairs, and training the graph neural network to distinguish the node-level relationships, obtaining the hierarchical association knowledge between event labels, and providing additional label hierarchical representations for text classification. Node classification and negative sampling can train the representation of the label graph by the graph neural network, learn the hierarchical relationship between labels, and enhance the text classification performance.

[0046] Among them, the hierarchical relationship refers to the superior-subordinate and belonging relationships existing between nodes in the graph, reflecting the positions of the nodes in the hierarchical structure. Adjacent nodes refer to the directly adjacent nodes connected to the central node in the graph. In this application, the hierarchical association relationship between event labels learned in the label graph. Nodes directly connected to the central label node in the label graph have a hierarchical association relationship. For each label, taking it as the central node, aggregating the representations of the adjacent nodes of the central node, where the adjacent nodes have a hierarchical relationship with the central node, to obtain the label semantic vector representation of the central node, reflecting the hierarchical association information between labels. Aggregating the representations of adjacent nodes with hierarchical associations can obtain an enhanced label semantic vector, providing additional context information for text classification.

[0047] Specifically, aggregating adjacent nodes with the label as the central node as the label vector, input: label graph G(V, E), where node V is the label and edge E is the hierarchical relationship, and the target label c is the central node. Graph neural network: includes an encoder to learn node representations and an aggregation module to aggregate adjacent nodes. Process: The encoder outputs the vector representations v of all nodes i , determining the vector v of the target label c c , finding the set of adjacent nodes N(c) directly connected to c in G, and for each node in N(c), aggregating its representation into v using an aggregation function c , obtaining the enhanced representation v' of the central node c . Aggregation function: mechanisms such as taking the average, maximum value, attention, etc., to fuse the representations of adjacent nodes into the central node. Finally, obtain the semantic vector v' of the central label c , including the hierarchical relationship information of its adjacent nodes. Repeat this process for all labels to obtain enhanced label representations.

[0048] Specifically, use a graph convolutional network to learn event label representations. Input layer: Input the event label set as graph nodes. Graph convolutional layer: Perform graph convolution operations on the label graph to learn the semantic relationships between label nodes and output node feature representations. Pooling layer: Perform pooling on the graph convolution output, using the maximum pooling algorithm to obtain the dimension-reduced node features. Fully connected layer: Perform a full connection on the pooling output to map the features into low-dimensional semantic vectors. Output layer: Finally, output the label semantic vector. Training process: The label semantic vector is used to assist text classification and is trained by backpropagation through the classification loss. Optimize the parameters of each layer to learn the label semantics. Finally, the graph convolutional network can learn the semantic relationships between event labels and output a vector representing the label semantics, providing additional context knowledge for the text classification task.

[0049] Preferably, the event labels in the text classification task are used as nodes, and the semantic relationships between the event labels are used as edges to construct a label graph; according to the label graph, node classification and negative sampling are performed through the graph attention network AgGNN to obtain the hierarchical relationships between the event labels; for each label in the event label set, in AgGNN, with the label as the central node, neighboring nodes with hierarchical relationships with the central node are aggregated through the attention mechanism as the label vector of the corresponding node; among them, AgGNN includes an input layer, a graph attention layer, a pooling layer, a fully connected layer, and an output layer: the input layer receives the event label set as input; the graph attention layer learns the semantic relationships between the event label sets through attention coefficients, and the attention coefficients are calculated through the features of the central node and neighboring nodes; the pooling layer reduces the dimensionality of the node features output by the graph attention layer through the average pooling algorithm; the fully connected layer maps the dimensionality-reduced node features into label vectors representing the semantics of the event labels; the output layer outputs the label vectors representing the label semantics.

[0050] Furthermore, through the attention mechanism of dynamic collaborative filtering, the text vector encoding and the label vector are fused to obtain the fusion feature, including: constructing a Transformer encoder as the text channel and constructing a graph convolutional network as the label channel; inputting the text vector encoding into the text channel, and mapping the text vector encoding to multiple subspaces through a fully connected layer to obtain multi-head text attention vectors; inputting the label vector into the label channel, and mapping the label vector to multiple subspaces through a fully connected layer to obtain multi-head label attention vectors; multiplying the obtained multi-head text attention vectors and multi-head label attention vectors element by element to obtain a gating vector representing the interactive attention of the text and the label; using the gating vector to perform weighted splicing on the text channel and the label channel to obtain the fusion feature of the text and the label.

[0051] Specifically, a text channel and a label channel are constructed. For the text channel: use a Transformer encoder, take the text sequence as input, and output the feature representation of the text. For the label channel: use a graph convolutional network, take the label set as input, and output the vector representation of the label. The Transformer encoder: includes multiple layers of self-attention and feed-forward networks to learn the context semantic information of the text. The graph convolutional network: includes graph convolutional layers, pooling layers, etc., to learn the semantic relationships between the labels. Finally, the following are obtained: for the text channel, the semantic representation of the text; for the label channel, the semantic representation of the label. The representations of the two channels can be concatenated for text classification or used separately as auxiliary information. End-to-end training is performed through the classification loss.

[0052] Specifically, the text vector encoding is input into the text channel to generate multi-head attention. The vector encoding of the input text is obtained, for example, through word embedding. A text channel is constructed, and attention structures such as Transformer are used. In the text channel, the text vector encoding is input. A multi-head attention mechanism is set, for example, 8-head attention. A linear transformation is performed on the input text vector: linear(x) = Wx + b, where x is the text vector, and W and b are learnable parameters. The result of the linear transformation is divided into multiple parts, and each part serves as the query, key, and value of one head of attention. Each head of attention calculates the attention weights according to the formula: The output of each head of attention is obtained as the multi-head attention vector. The multi-head attention vectors can be concatenated or averaged as the multi-head attention representation of the text. Backpropagation training is performed through the classification loss. The text vector enters the multi-head attention text channel, and through linear mapping and multi-head attention calculation, the multi-head attention representation of the text is obtained.

[0053] Among them, in this application, the vector representation of the input label is obtained, for example, through a graph convolutional network. A label channel is constructed, and a self-attention structure is used. In the label channel, the label vector is input. A multi-head attention mechanism is set, for example, 8-head attention. A linear transformation is performed on the input label vector: linear(x) = Wx + b, where x is the label vector, and W and b are learnable parameters. The result of the linear transformation is divided into multiple parts, and each part serves as the query, key, and value of one head of attention. Each head of attention calculates the attention weights according to the formula: The output of each head of attention is obtained as the multi-head label attention vector. The multi-head label attention vectors can be concatenated or averaged as the multi-head attention representation of the label. Backpropagation training is performed through the classification loss.

[0054] Specifically, the input is: the multi-head attention vector of the text: text_attn ∈ Rn×d, the multi-head attention vector of the label: label_attn ∈ Rm×d. Element-wise multiplication: text_attn and label_attn are multiplied element by element to obtain the gate vector: gate = text_attn ⊙ label_attn, and the dimension of gate is Rn×d. In implementation: text_attn and label_attn can be concatenated, and then a non-linear transformation is performed through a fully connected layer, and finally the gate vector is output. Gating mechanism: Each element in gate can be regarded as a gating unit, which is used to control the interaction information flow between the text and the label. Gating vector: It contains the interaction attention information between the text and the label, and can be used to obtain the correlation representation between the text and the label. The gating vector can be input into subsequent modules for text classification.

[0055] Specifically, gate vectors are used for text-label fusion. Inputs: text features output from the text channel: text_fea ∈ Rn×d, label features output from the label channel: label_fea ∈ Rm×d, and gate vectors: gate ∈ Rn×d. Gating operation: The gate vector is repeatedly concatenated to match the length of text_fea, and then element-wise multiplication is performed: text_gate = text_fea ⊙ gate. Weight fusion: text_gate and label_fea are concatenated, where text_gate plays a weighting role, to obtain the text-label fusion feature fused_fea. In implementation: Weighted sum fusion can also be performed, fused_fea = α × text_gate + β × label_fea, where α and β are learnable parameters. The fusion feature: contains the interaction information between the text and the label. By fully modeling the semantic relationship between the two, the fusion feature can be input into the classification module for text classification prediction.

[0056] Preferably, the steps of fusing the text vector encoding and the label vector through the attention mechanism include: constructing a Transformer encoder as the text channel and constructing a graph attention network AgGNN as the label channel; inputting the text vector encoding into the multi-head self-attention layer of the text channel, mapping the text vector encoding to multiple subspaces through linear transformation and concatenation to obtain multi-head text attention vectors; inputting the label vector into the graph attention layer of AgGNN, mapping the label vector to multiple subspaces through linear transformation and concatenation to obtain multi-head label attention vectors; adopting the co-attention mechanism to interact the multi-head text attention vectors and the multi-head label attention vectors, calculating the attention score matrices from the label to the text and from the text to the label through query vectors and key-value vectors; applying the attention score matrices to the multi-head text attention vectors and the multi-head label attention vectors, and obtaining the interacted text vector and label vector through weighted summation; concatenating the interacted text vector and label vector as the fusion feature.

[0057] Furthermore, obtaining the fusion feature of the text and the label includes: inputting the gate vector and the multi-head text attention vector into a fully connected layer to obtain the gated text feature; inputting the gate vector and the multi-head label attention vector into a fully connected layer to obtain the gated label feature; element-wise adding and connecting the gated text feature and the multi-head text attention vector to obtain the residual text feature; element-wise adding and connecting the gated label feature and the multi-head label attention vector to obtain the residual label feature; concatenating the residual text feature and the residual label feature as the fusion feature.

[0058] Specifically, obtain the gated text feature and the gated label feature. Input: gated vector: gate ∈ Rn×d, multi-head text attention vector: text_attn ∈ Rn×d, multi-head label attention vector: label_attn ∈ Rm×d. Obtain the gated text feature: Concatenate gate and text_attn. Input into the fully connected layer: text_gate_fea = FC(Concatenate[gate, text_attn]), to obtain the gated text feature text_gate_fea. Obtain the gated label feature: Concatenate gate and label_attn. Input into the fully connected layer: label_gate_fea = FC(Concatenate[gate, label_attn]), to obtain the gated label feature label_gate_fea. Implementation of the fully connected layer: It can be multi-layer non-linear transformation + dropout regularization. Learn the non-linear transformation to obtain new semantics. Finally, obtain: the gated text feature text_gate_fea, the gated label feature label_gate_fea. Both can be used alone or concatenated and input into the subsequent module for text classification.

[0059] Specifically, obtain the residual text feature. Input: gated text feature: text_gate_fea ∈ Rn×d, multi-head text attention vector: text_attn ∈ Rn×d. Element-wise addition: Add text_gate_fea and text_attn element-wise. Obtain the residual text feature: text_resid_fea = text_gate_fea + text_attn. Residual connection: The original attention feature text_attn transmits information through the residual structure to avoid the gating mechanism from destroying useful information. In implementation: It is also possible to concatenate the two and then connect to a fully connected layer to learn the non-linear transformation to generate the residual feature. The residual text feature: Contains both the original text attention information and the semantics information after gating regulation. The residual feature can be used alone or concatenated and input into the subsequent module for text classification.

[0060] Specifically, to obtain the residual label feature, the input is: gated label feature: label_gate_fea ∈ Rm×d, multi-head label attention vector: label_attn ∈ Rm×d. Element-wise addition: Add label_gate_fea and label_attn element-wise. Obtain the residual label feature: label_resid_fea = label_gate_fea + label_attn. Residual connection: The original attention feature label_attn transmits information through the residual structure. Avoid the gating mechanism from destroying useful information. In implementation: It is also possible to concatenate the two and then connect a fully connected layer to learn a non-linear transformation to generate the residual feature. The residual label feature: Contains both the original label attention information and the semantic information after gating regulation. The residual feature can be used alone or concatenated and input into subsequent modules for text classification.

[0061] Specifically, concatenate the residual text feature and the residual label feature as the fused feature. The input is: residual text feature: text_resid_fea ∈ Rn×d, residual label feature: label_resid_fea ∈ Rm×d. Concatenation operation: Concatenate text_resid_fea and label_resid_fea along the feature dimension to obtain the fused feature: fused_fea = concatenate(text_resid_fea, label_resid_fea). The dimension of the fused feature is R(n+m)×d. Concatenation method: It can be a simple vector concatenation or introduce weight parameters to weight the two. The fused feature: Contains the residual text feature and the residual label feature, fully combining the semantic information of the text and the label. The fused feature can be input into a classification module, such as a fully connected layer: output = FC(fused_fea), for text classification prediction.

[0062] Preferably, the steps to obtain the interacted text vector and label vector include: using the multi-head label attention vector as the query vector and the multi-head text attention vector as the key-value vector, and calculating the attention score matrix from the label to the text through scaled dot product; applying the attention score matrix from the label to the text to the multi-head text attention vector and obtaining the label-enhanced text vector through weighted summation; using the multi-head text attention vector as the query vector and the multi-head label attention vector as the key-value vector, and calculating the attention score matrix from the text to the label through scaled dot product; applying the attention score matrix from the text to the label to the multi-head label attention vector and obtaining the text-enhanced label vector through weighted summation; concatenating the label-enhanced text vector and the text-enhanced label vector as the interacted fused vector.

[0063] Furthermore, using the fused features as input, a neural network model for text classification is trained, including: constructing a multi-layer feed-forward fully-connected neural network as the text classification model; setting an input layer, where the number of neurons in the input layer is the same as the dimension of the fused features; setting multiple hidden layers, with each hidden layer using the Relu activation function; setting an output layer, where the number of neurons in the output layer is the same as the number of text classification categories, and using the softmax activation function; inputting the fused features into the constructed text classification model; and using the cross-entropy loss function to train the text classification model through the error backpropagation algorithm.

[0064] Among them, the feed-forward fully-connected neural network is the most basic neural network structure, consisting of an input layer, multiple hidden layers, and an output layer, with full connections between layers. There is no loop between each neuron, and information can only be propagated unidirectionally. In this application, a feed-forward fully-connected network containing multiple hidden layers is constructed as the discriminant model for text classification to classify and predict the text. Input layer: Receives the feature representation of the text. Hidden layer: Contains multiple fully-connected layers, performs non-linear transformation on the input, and extracts abstract features. Output layer: Outputs the text category prediction result, with full connections between layers and unidirectional information propagation, and is trained through backpropagation using the classification loss function.

[0065] Specifically, set the input layer. The input layer: As the first layer of the neural network, it receives external input. Input: already fused features: fused_fea ∈ Rn×d. Number of neurons in the input layer: Set the input layer to contain n neurons, where n is the same as the dimension n of the fused feature fused_fea. Implementation of the input layer: Define the input layer parameters: input_layer = tf.keras.layers.Input(shape=(n,)). n is the number of input neurons. Receive the input: Take fused_fea as the input of the input layer, input = input_layer(fused_fea). Output: The input layer does not perform transformation and directly outputs the input fused_fea, passing fused_fea to the next layer, and subsequent connections to hidden layers etc. for feature extraction and classification.

[0066] Among them, the Relu activation function is a non-linear activation function with the expression f(x) = max(0, x). It can introduce non-linear factors into the linear model and is often used in the hidden layer. The Softmax activation function can normalize a multi-dimensional vector into a probability distribution and is used in the output layer of multi-classification tasks. In this application, the Relu activation function is applied to the hidden layer to introduce non-linear transformation and increase the model's expressive power. The Softmax activation function is used in the output layer to convert the output into a probability representation for text classification prediction. The Relu activation function increases the non-linearity of the hidden layer; the Softmax activation function realizes multi-classification judgment. The collaborative application of the two constructs a text classification neural network model.

[0067] Among them, the cross-entropy loss function is often used in classification problems. It calculates the distance between the predicted probability and the true label to guide the model optimization. In this application, the cross-entropy loss function is used as the loss function of the text classification model. By calculating the cross-entropy between the predicted probability and the true label, it reflects the classification error. The fused feature fused_fea is input into the constructed text classification neural network. The model outputs the predicted probability distribution: pred_prob, and the true label: true_label (one-hot encoding). Calculate the cross-entropy loss: loss = Cross Entropy Loss(pred_prob, true_label). Perform gradient descent according to the loss loss, update the model parameters, minimize the loss, and optimize the text classification model. Perform gradient descent according to the loss loss, update the model parameters, minimize the loss, and optimize the text classification model.

[0068] Specifically, the cross-entropy loss function and error backpropagation are used to train the text classification model. For the cross-entropy loss function, define the cross-entropy loss function to calculate the distance between the predicted probability and the true label. The form of the loss function is: loss = -∑y * log(p), where y is the true label and p is the predicted probability. For error backpropagation, calculate the derivative of the loss with respect to the model parameters according to the loss function loss, and use the chain rule to propagate backward layer by layer to calculate the gradient of each parameter. Optimizers such as SGD update the parameters according to the gradient. For model training, initialize the model parameters, input the text features, output the predicted probability, calculate the loss - backpropagate - update the parameters, and repeat the above process until the loss converges. For implementation details, use the computational graph mechanism of the deep learning framework to complete backpropagation, and set hyperparameters such as the learning rate, number of iterations, and batch size. The cross-entropy loss function is suitable for classification tasks, and the backpropagation algorithm efficiently calculates the parameter gradients, realizing end-to-end training of the text classification model.

[0069] Preferably, the steps of training a text classification model using the fusion features include: constructing a multi-layer BiLSTM as the text classification model; inputting the fusion features into the first time step of the BiLSTM; at each time step, the BiLSTM separately uses a forward LSTM and a backward LSTM to learn the context information of the fusion features; at the last time step, the hidden states of the forward LSTM and the backward LSTM are concatenated and mapped to the dimension of the number of categories of text classification through a fully connected layer; setting an output layer, where the output layer uses a softmax activation function to output the probability distribution of each category; using a cross-entropy loss function and training the BiLSTM model through an Adam optimization algorithm to obtain the trained text classification model.

[0070] Compared with the prior art, the advantages of this application are as follows:

[0071] By using the pre-trained recurrent neural network XLNet to preprocess and extract features from the text, the limitations of the traditional bag-of-words model that cannot capture word order and long-distance dependencies are overcome, and the semantic features of the text are effectively learned. At the same time, by introducing sentence order prediction and word prediction tasks, the ability of XLNet to model text semantics is further enhanced.

[0072] The sentence order prediction task optimizes the model by randomly swapping the word order in the word order combination and using the loss between the predicted word order and the original word order, enabling XLNet to learn the impact of word order on text semantics. The word prediction task adopts a dual-channel structure and enhances XLNet's understanding of word semantics and context relationships by predicting masked words and learning the context representations of marked words.

[0073] The graph neural network is used to encode event labels. By constructing a label graph and using a graph convolutional network to learn the hierarchical relationships between labels, the problem that traditional one-hot label encoding cannot characterize the semantic relevance of labels is overcome. The graph convolutional layer aggregates neighbor node information, and the pooling layer and fully connected layer further refine the semantic representation of the labels, enabling the label vector to accurately capture the semantic features of event labels.

[0074] The attention mechanism is introduced. By constructing a text channel and a label channel, the interaction between the text vector and the label vector is realized using a gating mechanism. The gating vector weights the text and label features through element-wise multiplication, highlighting the text segments and key event labels important for the classification task. The introduction of residual connections further enhances the feature representation ability, enabling the fusion features to take into account both the original features and the interaction features.

[0075] Based on the fused features, a multi-layer feedforward fully connected neural network was constructed as a text classification model, and the Relu activation function and softmax activation function were used to achieve the non-linear transformation of the fused features and the calculation of classification probabilities. The use of the cross-entropy loss function and the error backpropagation algorithm ensured that the optimization process of the model could minimize the classification error and improve the classification accuracy. Brief Description of the Drawings

[0076] Figure 1 is an exemplary flowchart of a text classification method based on event tags shown in some embodiments of this specification;

[0077] Figure 2 is an exemplary flowchart of the pre-trained recurrent neural network XLNet shown in some embodiments of this specification;

[0078] Figure 3 is an exemplary flowchart of obtaining a loss function shown in some embodiments of this specification;

[0079] Figure 4 is an exemplary flowchart of obtaining fused features shown in some embodiments of this specification. Detailed Description of the Embodiments

[0080] The methods and systems provided in the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0081] Figure 1 is an exemplary flowchart of a text classification method based on event tags shown in some embodiments of this specification. Obtain the text, and perform preprocessing of word segmentation and stop word removal on the obtained text; use the pre-trained recurrent neural network XLNet to convert the preprocessed text into a sequence of word vectors; input the sequence of word vectors into the pre-trained Transformer encoder to output a text vector encoding representing the text semantics; according to the text classification task, construct an event tag set representing the text categories; use the tag set as the input, and use the pre-trained graph neural network to obtain the hierarchical relationship between the tags and output a tag vector representing the tag semantics; through the attention mechanism, fuse the text vector encoding and the tag vector to obtain the fused features; use the fused features as the input to train the neural network model for text classification; use the trained neural network model for text classification. By constructing a text channel to learn text semantics, constructing a tag channel to learn tag semantics, and effectively fusing the features of both, making full use of the complementary information of text and tag semantics, the performance of text classification can be improved.

[0082] Text preprocessing: Obtain the original text data, perform preprocessing such as word segmentation and stop word removal to get the preprocessed text. Obtain the original text data, collect the dataset for the text classification task, such as the movie review dataset. Each piece of data contains a text segment and the corresponding classification label. Example of the original text: "The movie is amazing and inspiring, I really love it.". Word segmentation: Segment the original text, for example, using the Spaces word segmentation tool. After segmentation, the text is: "The movie is amazing and inspiring, I really love it.". Stop word removal: Remove some stop words that have no practical meaning. Stop word list: ["is", "and", "the", "it"]. After stop word removal, the text is: "movie amazing inspiring, I really love.". Normalization: Convert to lowercase uniformly, standardize punctuation, etc. After preprocessing, the text is: "movie amazing inspiring, i really love.". Preprocess the original text through word segmentation, stop word removal, etc. to obtain clean and normalized text data for subsequent processing. Preprocessing is an important step in the text classification task.

[0083] Text semantic extraction: Collect the dataset for the text classification task, such as data containing movie reviews. The dataset contains N = 5000 text samples. Each sample contains a text segment and the corresponding classification label. For example, text sample T1: Text: "The movie is amazing and inspiring, it is definitely one of the best movies I have ever seen.". Label: Positive. Text sample T2: Text: "I don't like the plot of this movie, it is boring and not logical at all. A waste of time.", Label: Negative,... and so on to construct N text samples. Represent all samples as: T = {T1, T2,..., TN}, where Ti represents the i-th text sample. This sample set can be used to pre-train the XLNet model to learn the text semantic feature representation. It is very important to construct a large-scale sample set with high quality and containing semantic labels. Thus, construct the set T containing N = 5000 labeled text samples, each sample containing a text segment, which can be used in the pre-training process of XLNet.

[0084] Figure 2It is an exemplary flowchart of the pre-trained Transformer-XL (XLNet) according to some embodiments of this specification. XLNet includes two main components: a word vector representation layer and an L-layer bidirectional Transformer encoder. Word vector representation layer: Input a sequence of text words, look up the word vectors of each word (pre-trained based on models such as BERT, etc.), and convert each word into a word vector with a fixed dimension, such as 256 dimensions. L-layer bidirectional Transformer encoder: Input a sequence of word vectors, set the encoder to include L = 12 layers, each layer includes a multi-head self-attention mechanism and a feed-forward fully-connected network, and the layers are composed of residual connections and LayerNorm to implement a bidirectional encoder. Example of model structure: Word vector representation layer, bidirectional Transformer encoder: Encoder layer 1, Encoder layer 2,..., Encoder layer L, output a text semantic feature vector. By constructing an XLNet model containing an L-layer bidirectional Transformer encoder, the semantic feature representation of the text can be learned for downstream tasks. A deep network can extract semantic information at different granularities. Given a text sample Ti: "The movie is amazing and inspiring", factorize Ti to generate m word order combinations: m = 3, {wi1} = "The movie is amazing", {wi2} = "is amazing and", {wi3} = "amazing and inspiring", and permute these 3 word order combinations to obtain 3 semantic combinations: {si1} = "The movie is amazing", {si2} = "is amazing and inspiring", {si3} = "amazing and inspiring The movie", where each semantic combination represents an order permutation of the sample text. Through permutation and combination, a text sample can generate multiple training samples. This increases the amount of training data and improves the generalization ability of the model. m can vary according to the text length.

[0085] Figure 3 It is an exemplary flowchart of obtaining a loss function according to some embodiments of this specification. Set two training objectives for XLNet. Sentence order prediction task T1: Objective, predict the original sentence word order. Input: A sentence with shuffled word order {w' i}, Output: The predicted probability {p(w i )} of the original word order. Loss function: -λ log p(w i ) - (1 - λ)C. Word prediction task T2: Objective, predict the semantics of the masked word and the token word. Input: The masked sentence {T'} to channel 1, the original sentence + the token word {w m} to Channel 2, Output: The probability p of the masked word predicted by Channel 1 i , the context c of the token word predicted by Channel 2 i . Loss function: loss2 = loss 21 (p i ) + λloss 22 (c i ). Overall objective: Predict the word order through T1 to learn the sentence semantics, predict the words through T2 to learn the word semantics. The two objectives provide sentence-level and word-level supervision to comprehensively learn the text semantics. In this way, by setting two training objectives of order prediction and word prediction, the semantic features of the text can be supervised at both the sentence and word levels for XLNet to learn.

[0086] Training process of the sentence order prediction task T1: The original sentence word order {w i}: "The movie is amazing", swap the word positions to get the new word order {w' i}: "is amazing The movie", input {wi'} into XLNet to predict the original word order {p(w i )}, assume the prediction probability: p(w i ) = 0.7.

[0087] Calculate the prediction loss: loss = -λlog(0.7) - (1 - λ)C, where λ is the balance factor, for example, take 0.5, D is a constant term, for example, take 1. Get the loss: -0.5log(0.7) - 0.5×1 = 0.35. Use the gradient of the loss value to update the XLNet model parameters through backpropagation. Repeat the above process, and XLNet gradually learns to reconstruct the original sentence order, thereby capturing the context semantic information of the text. Training process of the word prediction task T2, the original sentence {w i}: "The movie is amazing and inspiring", mask the words to get {T'}: "The movie is____and_____", input {T'} into Channel 1 to predict the probability pi of the masked words. Tokenize the words to get {w m}: "The movie is amazing and inspiring", input {w i} and {w m} into Channel 2 to get the context representation c i . Calculate the word prediction loss: loss 21 (p i ) is the masked word prediction loss, loss 22 (c i ) is the token word context loss, and λ is the hyperparameter weight.

[0088] Loss function: loss2 = loss 21 (p i ) + λloss 22 (c i ). Perform gradient descent according to loss2 to update the XLNet parameters, and repeat this process. XLNet learns to predict words and context semantics. Use the pre-trained XLNet for text semantic encoding. Train XLNet on a large-scale text dataset. After sufficient iterations, obtain the pre-trained XLNet model. For the new sample text T: "The movie is amazing and inspiring", input the text T into the pre-trained XLNet to generate a sequence of word vectors: [word_vec1, word_vec2,..., word_vecN]. Input the sequence of word vectors into the pre-trained Transformer encoder. The Transformer encoder contains multiple layers of self-attention and feed-forward networks. Through hierarchical encoding, obtain the semantic feature vector encoding of the text T, text_encoding ∈ Rd, where d is the vector dimension, such as d = 1024. text_encoding contains the semantic information of the text T and can be input into downstream text classification and other tasks to extract higher-quality text semantic features with the help of the pre-trained XLNet.

[0089] Label semantic extraction. The text classification task contains 4 categories: Node 1: Positive, Node 2: Negative, Node 3: Neutral, Node 4: Question. Construct a label graph: 4 nodes representing 4 category labels. Set the edges to represent the relationships between the labels: Edge 1-2: Positive and negative are opposite relationships, Edge 2-3: Negative and neutral are similar relationships, Edge 1-4: Positive and question have no specific relationship, Edge 3-4: Neutral and question are similar relationships. Label graph representation: Positive - Negative, Neutral - Question. The graph contains node features and edge features. Learn the node semantics and relationships through node classification and negative sampling, and finally obtain the vectors representing the semantics of each node (label). Construct a graph convolutional network structure. Input layer: Input dimension: The number of labels, such as 4 labels. Graph convolutional layer: Input: Node feature matrix, edge relationship matrix, learn node semantics and edge relationships, output an aggregated feature matrix. Pooling layer: Perform max pooling on the output of the graph convolutional layer to reduce the number of parameters and prevent overfitting. Fully connected layer: Input the pooled features and map them to low-dimensional semantic vectors. The fully connected network realizes non-linear transformation. Output layer: Output a label semantic vector with a length of 128. Network structure schematic: Input layer - Graph convolutional layer - Pooling layer - Fully connected layer - Output layer.

[0090] Graph Convolutional Network Training Process, Node Classification Training: Input nodes and edge relationships, predict the category of each node, and perform supervised classification training. Negative Sampling: Randomly sample incorrect node pairs to increase the contrast between positive and negative samples and improve the model's discrimination ability. Aggregate neighboring nodes: For the central node, aggregate the features of neighboring nodes, learn the local semantic information on the node, and output the semantic vector of the central node. Training Steps: Input the labeled graph, perform node classification prediction + negative sampling, and graph convolution to aggregate neighboring features. Calculate the loss and update the parameters. Training Objective: Learn node semantics and relationships. Through mechanisms such as node classification, negative sampling, and feature aggregation, the graph convolutional network can learn the node semantics and relationships in the labeled graph and achieve a vectorized representation of label semantics.

[0091] Using the graph convolutional network to output the label semantic vector, input the label "Positive". In the graph convolutional network, take "Positive" as the central node. Aggregate the features of relevant neighboring nodes of the central node: "Negative": Opposite node, "Neutral": Similar node. After network operations, output a vector v1 with a length of 128. Input the label "Negative", take "Negative" as the central node, aggregate relevant nodes: "Positive": Opposite node, "Neutral": Similar node, and output a vector v2 with a length of 128. Vector v1 represents the semantics of "Positive", and vector v2 represents the semantics of "Negative". The vector with a length of 128 carries the semantic information of the label. In this way, for each label, the graph convolutional network can output a fixed-length vector representing its semantics, providing semantic input for subsequent text classification tasks. The label semantic vector represents the semantic information of each category: "Positive" -> Positive semantic vector (length 128), "Negative" -> Negative semantic vector (length 128), "Neutral" -> Neutral semantic vector (length 128). Input the semantic vector into the text classification model: Text Encoder part: Extract text semantic features, Label Encoder part: Use the graph convolutional network to output the label vector. In the classification model: Concatenate the text semantic features and the label semantic vector, obtain the text-label matching score through the fully connected layer, calculate the loss using cross-entropy, and train the model.

[0092] Figure 4It is an exemplary flowchart for obtaining fused features shown in some embodiments of this specification. Text-label fusion is performed to construct a text channel and a label channel. For the text channel - Transformer encoder, the input is the text vector encoding text_encoding. The encoder contains multiple self-attention layers, and the output is the text feature vector text_features. For the label channel - graph convolutional network, the input is the label vector encoding label_encoding, which includes graph convolutional layers, pooling layers, etc., and the output is the label feature vector label_features. The two-channel network structure is as follows: for the text channel, text vector -> Transformer encoder -> text features; for the label channel, label vector -> graph convolutional network -> label features. The text channel learns text semantics, and the label channel learns label semantics. Subsequently, the features of the two channels are fused through multi-head attention. In this way, the text channel and the label channel are constructed to learn the semantic features of text and labels respectively, preparing for the subsequent multi-head attention fusion. Text encoding: text_encoding = [0.1, 0.3, 0.8,...], a vector of length 1024, and label encoding: label_encoding = [0.6, 0.1, 0.4,...], a vector of length 128. They are input into the text channel: Transformer encoder, with the input being text_encoding, and into the label channel: graph convolutional network, with the input being label_encoding. In the text channel, text_encoding represents the semantic features of the text. In the label channel, label_encoding represents the semantic features of the label. The text channel and the label channel respectively learn their own semantic representations. Subsequently, the features of the two channels will be fused through the multi-head attention mechanism. In this way, the text encoding text_encoding serves as the input to the text channel, representing text semantics; the label encoding label_encoding serves as the input to the label channel, representing label semantics.

[0093] Generate multi-head attention in the text channel and the label channel. In the text channel: The text_encoding is input into a fully connected layer, which is mapped to 8 subspaces to obtain 8 vectors: [text_head1, text_head2,..., text_head8], namely the multi-head text attention vectors: multi_head_text. In the label channel: The label_encoding is input into a fully connected layer, which is mapped to 8 subspaces to obtain 8 vectors: [label_head1, abel_head2,..., label_head8], namely the multi-head label attention vectors: multi_head_label. The role of multi-head attention: Express semantics in different subspaces, learn semantics from different perspectives, and improve the expression ability of the model. Subsequently, two groups of multi-head attention vectors will be used to calculate the interactive attention between text and label. In this way, the text and label features are mapped to multiple subspaces through the fully connected layer, obtain as multi-head attention, and prepare for calculating the dynamic interactive attention between text and label. Calculate the gating vector. Given the multi-head text attention vectors: multi_head_text = [text_head1, text_head2,..., text_head8], and the multi-head label attention vectors: multi_head_label = [label_head1, label_head2,..., label_head8], perform element-wise multiplication: gate1 = text_head1 ⊙ label_head1,

[0094] gate2 = text_head2 ⊙ label_head2... gate8 = text_head8 ⊙ label_head8. Concatenate the results: gate = [gate1, gate2,..., gate8]. gate represents the interactive attention between each part of the text and each part of the label. gate will be used to weight and fuse the features of the text channel and the label channel, and dynamically learn the key interactive regions between text and label. In this way, through the element-wise multiplication of the multi-head attention vectors, the dynamic interactive attention gating vector between text and label is obtained, preparing for the weighted fusion of features.

[0095] Obtain the gated text feature. Given the gated vector: gate, and the multi-head text attention vector: multi_head_text, concatenate the two into a new vector: [gate, multi_head_text], and input it into the fully connected layer for non-linear transformation: FC([gate, multi_head_text]). The parameters of the fully connected layer are: input dimension: the dimension of the gate vector + the dimension of the multi_head_text vector, output dimension: the dimension of the gate_text vector. Obtain the gated text feature: gate_text = FC([gate, multi_head_text]). gate_text fuses the gated vector and the text attention, representing the gated text feature. In this way, the gated vector gate and the text feature multi_head_text are non-linearly transformed through the fully connected layer to obtain the gated text feature gate_text. Obtain the gated label feature gate_label. Given the gated vector: gate, and the multi-head label attention vector: multi_head_label, concatenate the two into a new vector: [gate, multi_head_label], and input it into the fully connected layer for non-linear transformation: FC([gate, multi_head_label]). The parameters of the fully connected layer are: input dimension: the dimension of the gate vector + the dimension of the multi_head_label vector, output dimension: the dimension of the gate_label vector. Obtain the gated label feature: gate_label = FC([gate, multi_head_label]). gate_label fuses the gated vector and the label attention, representing the gated label feature. In this way, the gated vector gate and the label feature multi_head_label are non-linearly transformed through the fully connected layer to obtain the gated label feature gate_label.

[0096] Generate residual text features and residual label features. Given the gated text feature: gate_text, the multi-head text attention vector: multi_head_text, generate the residual text feature: residual_text = gate_text + multi_head_text. The residual connection can preserve the original attention information. Given the gated label feature: gate_label, the multi-head label attention vector: multi_head_label, generate the residual label feature: residual_label = gate_label + multi_head_label, which also preserves the original label attention information. residual_text and residual_label respectively fuse the gated features and the original attention features as enhanced representations of the text features and label features. In this way, by performing residual addition on the gated features and the original attention features, residual text features and residual label features are obtained, realizing the fusion of multi-head attention. Finally, the text-label fusion feature is obtained. Given the residual text feature: residual_text, the residual label feature: residual_label, concatenate the two: fusion_feature = Concat(residual_text, residual_label), where: Concat represents the concatenation operation, and fusion_feature is the text-label fusion feature, which contains: the residual text feature residual_text, the residual label feature residual_label, fusing text attention and label attention, fully modeling the interaction between text and label. fusion_feature can be used as an enhanced feature for text classification. In this way, by concatenating the residual text feature and the residual label feature, the multi-head attention fusion feature of text and label can be finally obtained as the semantic feature input for text classification.

[0097] Fine-tune the classification model and construct a multi-layer fully connected network. Input layer: Input dimension: The same as the dimension of the fusion feature fusion_feature, for example, 256. Hidden layer 1: Number of neurons: 1024, Activation function: Relu. Hidden layer 2: Number of neurons: 512, Activation function: Relu. Output layer: Number of neurons: The number of classes, for example, 4, Activation function: Softmax. Network structure: Input layer (256 neurons), Fully connected layer 1 (1024 neurons, Relu), Fully connected layer 2 (512 neurons, Relu). Output layer (4 neurons, Softmax). In this way, a multi-layer neural network containing an input layer, fully connected hidden layers, and an output layer is constructed as the basic structure of the text classification model. Take the fusion feature as the input and calculate the cross-entropy loss. Input the fusion feature fusion_feature: The dimension of fusion_feature is 256. Input it into the input layer of the classification model: The input layer contains 256 neurons, and the input value is set to fusion_feature. Calculate the cross-entropy loss in the output layer: The output layer contains 4 neurons, each representing a class. Calculate the cross-entropy between the output layer vector and the one-hot encoding of the true class. Obtain the loss loss of the sample. Cross-entropy calculation formula: loss = -sum(y_true * log(y_pred)), where y_true is the true class and y_pred is the model output probability. Minimize the loss to optimize the model parameters. In this way, input the fusion feature into the classification model, calculate the error between the model output and the true label through the cross-entropy loss function, and optimize the model. Use the error backpropagation algorithm to Fine-tune the text classification model. The cross-entropy loss loss has been calculated. Use the loss loss for error backpropagation: Calculate the gradient of the loss with respect to the model parameters and update the parameters using the gradient descent method. Backpropagation process: Start from the output layer, calculate the parameter gradients layer by layer, use the chain rule to calculate complex gradients, and update each parameter. Use the Fine-tuned text classification model to predict new texts. For the input new text: Generate text features text_features through the text channel and label features label_features through the label channel. Fusion of text features and label features: Adopt the same multi-head attention mechanism to obtain a new fusion feature fusion_feature. Input fusion_feature into the Fine-tuned classification model: After processing by the input layer and hidden layers, obtain the predicted class distribution in the output layer. Select the class corresponding to the maximum value in the output layer distribution: For example, [0.1, 0.2, 0.6, 0.1], select the class corresponding to index 2, with the highest probability, to obtain the prediction result of the model for the new text and complete the text classification based on the fusion feature.Data preparation: Collect text data and corresponding category labels. Text data preprocessing: Perform operations such as word segmentation and stop word removal to generate text semantic encodings. Build a text channel: Use semantic encoding models such as Transformer. Build a label channel: Use a graph convolutional network to learn label Embeddings. Text-label multi-head attention fusion: Generate fused features. Build a classification model: A multi-layer fully connected neural network. Input the fused features into the classification model for training to obtain a fine-tuned text classification model. For new text, repeatedly generate fused features, input the features into the classification model, and obtain classification prediction results, ultimately realizing text classification based on text-label fusion attention.

Claims

1. A text classification method based on event tags, comprising: Obtain a text, and perform preprocessing of word segmentation and stop word removal on the obtained text to obtain the preprocessed text; Utilize the pre-trained Recurrent Neural Network with Transformer XLNet to convert the preprocessed text into a sequence of word vectors; Input the obtained sequence of word vectors into the pre-trained Transformer encoder, and output a text vector encoding representing the semantics of the text; According to the text classification task, construct an event tag set representing text categories; Use the event tag set as input, and utilize the pre-trained graph neural network to obtain the hierarchical relationship between tags, and output a tag vector representing the semantics of the tags, including: taking the event tags in the text classification task as nodes and the semantic relationships between event tags as edges to construct a tag graph; according to the tag graph, perform node classification and negative sampling through the Graph Attention Network AgGNN to obtain the hierarchical relationship between event tags; for each tag in the event tag set, in AgGNN, take the tag as the central node, and aggregate the neighboring nodes with hierarchical relationships with the central node through the attention mechanism as the tag vector corresponding to the node; Through the attention mechanism, fuse the text vector encoding and the tag vector to obtain a fused feature, including: input the text vector encoding into the multi-head self-attention layer of the text channel, map the text vector encoding to multiple subspaces through linear transformation and splicing to obtain a multi-head text attention vector; input the tag vector into the graph attention layer of AgGNN, map the tag vector to multiple subspaces through linear transformation and splicing to obtain a multi-head tag attention vector; adopt the co-attention mechanism to interact the multi-head text attention vector and the multi-head tag attention vector, calculate the attention score matrices of tag-to-text and text-to-tag through query vectors and key-value vectors; apply the attention score matrices to the multi-head text attention vector and the multi-head tag attention vector, and obtain the interacted text vector and tag vector through weighted summation; splice the interacted text vector and tag vector as the fused feature; Use the fused feature as input to train a neural network model for text classification; Use the trained neural network model for text classification to classify the preprocessed text to obtain a text classification result.

2. The text classification method based on event tags according to claim 1, wherein: Obtain the pre-trained Recurrent Neural Network with Transformer XLNet, including: Construct text samples ; Construct a Recurrent Neural Network with Transformer XLNet containing multiple layers of bidirectional Transformer encoders for learning the semantic features of text samples; Input text sample , decompose the text sample using the factorization algorithm , generate m word order combinations of the text sample ; The generated m word order combinations are permuted and combined to obtain m semantic combinations ; Set the training objectives of the recurrent neural network XLNet, where the training objectives include a sentence order prediction task and a word prediction task ; Utilize the text sample {T} and the set training objective to train the Recurrent Neural Network with Transformer XLNet to obtain the pre-trained Recurrent Neural Network with Transformer XLNet.

3. The text classification method based on event tags according to claim 2, wherein: Sentence order prediction task , including: Obtain m word order combinations ; Randomly select the positions of n words in the and swap the order of the words. Repeat the swapping A times to obtain A combinations of word orders. ; According to A word order combinations , using the recurrent neural network XLNet, output A predicted word orders ; Calculate the word order of A predictions with the corresponding word order combinations the loss between , as the error; According to the loss , update the parameters of the recurrent neural network XLNet through the gradient descent algorithm.

4. The text classification method based on event tags according to claim 3, wherein: Calculate the loss : wherein, is the original word order output by the model of the predicted probability; is a coefficient, with a value range of 0 to 1; D is a constant term.

5. The text classification method based on event tags according to claim 4, wherein: The word prediction task, including: Obtain m word order combinations ; Construct a dual-channel neural network, including a first channel and a second channel; Randomly select B words from the word order combination for masking processing to generate masked samples ; Take the masked sample as input and feed it into the first channel to output the prediction result for the masked word ; Randomly select C words from the word order combination for marking to obtain marked words ; Combine the word order and the marked words as inputs and input them into the second channel, and output the marked words The corresponding context semantic representation in the combined word order ; Calculate the prediction loss of words : Among them, is the prediction loss of the first channel, is the context loss of the second channel; According to Update the parameters of the recurrent neural network XLNet through the gradient descent algorithm.

6. The text classification method based on event tags according to claim 5, characterized in that: The prediction loss for the first channel: Among them, y is the true one-hot label of B masked words; is the predicted probability of the first-channel model for these B words; represents a parameter, represents a parameter is the L2 regularization term of; represents the regularization term weight hyperparameter; Context loss for the second channel: Among them, represents the C token word context representations output by the second channel; represents the true context representation corresponding to the token word; represents and the distance function between; represents and the similarity function of; and are loss term weight hyperparameters.

7. The text classification method based on event tags according to claim 1, characterized in that: Obtain the fusion features of the text and the tags, including: Input the gating vector and the multi-head text attention vector into the fully connected layer to obtain the gated text features; Input the gating vector and the multi-head label attention vector into the fully connected layer to obtain the gated label features; Element-wise add the gated text features and the multi-head text attention vector to obtain the residual text features; Element-wise add the gated label features and the multi-head label attention vector to obtain the residual label features; Concatenate the residual text features and the residual label features as the fusion features.

8. The text classification method based on event tags according to claim 7, characterized in that: Use the fusion features as the input to train the neural network model for text classification, including: Construct a multi-layer feedforward fully connected neural network as the text classification model; Set the input layer, and the number of neurons in the input layer is the same as the dimension of the fusion features; Set multiple hidden layers, and each hidden layer uses the Relu activation function; Set the output layer, and the number of neurons in the output layer is the same as the number of text classification categories, and the softmax activation function is used; Input the fusion features into the constructed text classification model; Adopt the cross-entropy loss function and train the text classification model through the error backpropagation algorithm.

Images