Intelligent official document classification system based on label semantic comparison fusion perception

Abstract

The invention discloses an intelligent official document classification system based on label semantic comparison fusion perception, which can adaptively learn and adapt to different types of official document data, and does not depend on a fixed mode rule, thereby improving the processing capability of diversified data. By adopting an advanced artificial intelligence classification algorithm, the understanding and classification precision of the system on official document contents is improved, and particularly, when documents with similar label semantics are processed, nuisance can be accurately distinguished; the system can automatically learn and optimize a classification strategy, reduces manual intervention, and improves the working efficiency and the processing speed.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent document classification, and in particular relates to an intelligent document classification system based on label semantic comparison and fusion perception. Background Art

[0002] In the field of intelligent document classification, the present invention has strict specifications and requirements for the processing and management of official documents. Corresponding official documents cover different responsibility mappings and need to be classified and processed and distributed according to corresponding categories. On the one hand, the traditional method requires manual review and judgment of the document number, source unit and title of the official document one by one. This time-consuming and labor-intensive process is not only inefficient, but also prone to omissions. At the same time, the intervention of human subjective factors may interfere with the accuracy of official document classification, making it difficult for the classification results to quickly adapt to business changes, requiring a lot of manual adjustments.

[0003] On the other hand, although other existing document classification methods try to use artificial intelligence technologies, such as recurrent neural networks (RNN) and bidirectional long short-term memory networks (BILSTM), to model text semantics, these methods have some problems when faced with huge amounts of data. First, in terms of accuracy, as the amount of data increases, the accuracy of the prediction results may decrease because existing methods may not be able to handle the complexity of large-scale data. Secondly, semantic confusion is also a challenge. The labels of some documents may be semantically ambiguous, resulting in unsatisfactory classification results. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent official document classification system based on label semantic contrast and fusion perception, and to develop a new algorithm or model. The system can adaptively learn and adapt to different types of official document data without relying on fixed pattern rules, thereby improving the processing capabilities of diversified data; by adopting advanced artificial intelligence classification algorithms, the system's understanding of the content of official documents and classification accuracy are improved, especially when processing documents with similar label semantics, the subtle differences can be accurately distinguished; the system can automatically learn and optimize classification strategies, reduce manual intervention, and improve work efficiency and processing speed.

[0005] The technical solution of the present invention:

[0006] An intelligent document classification system based on label semantic comparison and fusion perception, the system comprising:

[0007] The text representation module is used to map each word in the input sentence into a high-dimensional vector space and use BILSTM to capture the contextual semantic relationship between words to obtain the contextual semantic representation of the sentence;

[0008] Attention mechanism module, including self-attention mechanism and label semantic attention mechanism. The self-attention mechanism is used to calculate the importance of each word in a sentence and generate attention weights to obtain a comprehensive contextual semantic representation of the sentence. The label semantic attention mechanism is used to calculate the importance of each word for each responsibility label and generate contextual semantic representations related to the label.

[0009] The contrast fusion module is used to analyze the difference between the comprehensive contextual semantic representation of the sentence obtained by the self-attention mechanism and the contextual semantic representation related to the label obtained by the label semantic attention mechanism to obtain the loss; the difference between the contextual semantics of the sentence itself obtained by the self-attention mechanism and the label-related information obtained by the label semantic attention mechanism is used for contrast learning and feedback to obtain the contrast loss; and the cross entropy loss function is used to add weights to obtain the difference between the probability distribution of the model prediction and the probability distribution of the real label, which is used as a basis for improving the classification accuracy of the model;

[0010] The label prediction module is used to adaptively weight the comprehensive contextual semantic representation of the sentence obtained by the self-attention mechanism and the contextual semantic representation related to the label obtained by the label semantic attention mechanism, and combine the model prediction probability distribution of the contrast fusion module and the probability distribution difference of the true label to obtain the final sentence feature representation. The final sentence feature representation is input into the classifier to output the probability of predicting each responsibility label, and the final predicted responsibility label can be determined by setting an appropriate threshold.

[0011] Furthermore, the text representation module includes two submodules: pre-training and BILSTM. The pre-training submodule is used to convert the text into a trainable vector representation, and the BILSTM submodule is used to extract semantic information in the sentence.

[0012] The pre-training submodule uses the ALbert pre-training model trained based on a large-scale text corpus to map the input text sequence to the pre-trained word embedding space to generate a high-dimensional text representation containing semantic information;

[0013] Given a sentence in a document, its word sequence W = {w1, w2, ..., w n}, n represents the number of words in the sentence, and the vector representation of the words in the document is obtained after the pre-training module: W′={w′1,w′2,...,w′ n},w i Represents the i-th word in the sentence, which is expressed as follows:

[0014] w′ i = Albert(w i )

[0015] The BILSTM submodule is used to process the words in the sentence forward and backward, obtain the hidden states of the sentence in both the forward and reverse directions, and obtain the context information in the text sequence;

[0016] The vector representation W′ of the text passes through the LSTM module and gets two inputs through the forward and reverse sentence sequences respectively:

[0017]

[0018] in, and h i is its corresponding forward / reverse word representation, so the entire document can be represented as:

[0019]

[0020] in, Represents text in two directions.

[0021] Furthermore, the specific steps of the self-attention mechanism are as follows:

[0022] First, the query, key, and value are calculated. For each element in the input sequence, a query vector, a key vector, and a value vector are calculated. These vectors are obtained by linearly transforming the learned weight matrix and the input elements. The input is a unidirectional sequence H = {h1, h2, ..., h n}, the weight matrix is ​​W Q , W K , W V , then for each element h t ,have:

[0023] Q i =W Q ·h i

[0024] K t =W K ·h i

[0025] V i =W V ·h i

[0026] Among them, Q i , K t , V i Represent the query, key, and value vectors of the i-th element respectively;

[0027] Next, the attention score is calculated, for each pair of elements h i and h j , an attention score is calculated, indicating hi For h j The attention score is calculated by taking the dot product of the query vector and the key vector and then dividing it by a scaling factor:

[0028] where d k is the dimension of the key vector;

[0029] Then the attention weight is calculated, and the softmax function is used to convert the attention score into a value between 0 and 1 and the sum is 1, thereby obtaining the attention weight:

[0030] a ij =softmax(score(Q i , K j ))

[0031] Finally, calculate the output, multiply the value vector of each element by its corresponding attention weight, and then sum them up to get the final output:

[0032] In the above formula, V j represents the value vector of each element, m i Represents the generated new context-related word vector, which contains the information of the current word and its context. Finally, all the word vectors form the sentence vector M s Has contextual meaning.

[0033] Furthermore, the specific steps of the label semantic attention mechanism are as follows: first, obtain the label semantic vector representation:

[0034] For a set of label texts E, the text content of any label is represented by e = {w1, w2, ..., w p}, p is the length of the text label, which is usually between 1 and 3.

[0035] In order to obtain the representation c of each label i , using the text content of the label as input, first obtain the vector representation of each word through the pre-trained model, and then use the word vector average function to calculate the vector representation of the label, as described below:

[0036]

[0037] All the labels in the dataset are represented as a matrix C. Next, the word-label correlation is calculated. An attention mechanism based on label semantic information is designed to obtain the importance of each word. The weight of each word to the current label is obtained by calculating the matching score between the word in the document and each label:

[0038] a=CHT

[0039] Get the matching scores of all tags and words:

[0040]

[0041] Get the matching score of all tags for each word in the current document

[0042] Finally, the output is calculated. From the matching score, we can get the part of the document that each tag pays more attention to, and then combine the text vector representation to get the document representation corresponding to the tag:

[0043]

[0044] Here is the feature that each label pays more attention to in the forward and reverse sentence representations of the document, and finally M l The document representation corresponding to all tags.

[0045] Furthermore, the comparison and fusion module comprises the following specific steps:

[0046] First, the text representation vectors obtained by the self-attention mechanism and the label semantic attention mechanism are normalized by softmax:

[0047] y (l) =softmax(M (l) )

[0048] y (s) =softmax(M (s) )

[0049] Secondly, in order to make the prediction model fully utilize this information, a KL divergence method is used for comparison, which can be expressed as follows:

[0050] Loss(KL)=KL-divergence(y (s) ,y (l) )

[0051] Then, the model itself uses the cross entropy loss function to update the model. The formula is as follows:

[0052]

[0053] where y ij is a binary value used to identify whether text i belongs to tag j, where n represents the total number of words and l represents the total number of tags;

[0054] Finally, add the two losses to get the final Loss representation:

[0055] Loss = aLoss(KL)+L

[0056] Where α is the weight of label confusion comparison loss.

[0057] Furthermore, the label prediction module is divided into two sub-modules: adaptive attention fusion and classification. The adaptive attention fusion sub-module combines the features extracted by the self-attention mechanism and the label-based label semantic attention mechanism to obtain the final text representation. The classification sub-module classifies the text to obtain the probability value of the text for each label.

[0058] Furthermore, the adaptive attention fusion part uses an adaptive method to fuse the features extracted by the content-based self-attention mechanism and the tag-based label semantic attention mechanism, and selects the more important part from the features extracted by the above two attentions to generate a vector representation that is more representative of the document. Specifically, two weight vectors are used to obtain the importance of the above two parts of the document representation, which can be obtained through a fully connected layer:

[0059] α=sigmoid(M (s) W1)

[0060] β=sigmoid(M (l) W2)

[0061] Among them, W1 and W2 are parameters, which represent the importance of self-attention mechanism and label semantic attention mechanism respectively. So add restrictions to these two pairs of parameters:

[0062] α j +β j =1

[0063] Get the final document representation:

[0064] M=αM (s) +βM (l)

[0065] Furthermore, in the classification submodule, after obtaining the final document representation, a multi-label text classifier is established through two fully connected layers, and the predicted probability of each label can be obtained by the following formula:

[0066]

[0067] Among them, W3 and W4 are parameters, representing the weight matrix of the fully connected layer. The value output by sigmoid can be regarded as probability. What is obtained here is the probability of the responsibility label corresponding to each document. The responsibility labels exceeding the threshold are retained, and the corresponding departments and leaders are found through the responsibility-department leader table to obtain the final prediction result.

[0068] The advantages of the present invention may be:

[0069] 1. By introducing the contrastive fusion perception method and the application of attention mechanism and contrastive learning technology, it is possible to more accurately capture the association between the contextual semantics of the sentence and the label semantics, thereby improving the accuracy of classification. This method can effectively solve the label confusion problem and make the classification results more accurate and reliable.

[0070] 2. The method of the present invention fully considers the label semantic information in the classification process. By introducing the label semantic attention mechanism, the importance of each word for each label can be calculated, and the contextual semantic representation related to the label can be generated. Such a design enables the classification model to better understand the semantic relationship between text and label, and improve the accuracy and robustness of classification.

[0071] 3. The contrastive learning module is one of the key components of the present invention. This module can analyze the difference between the contextual semantics of the sentence itself and the label-related information, and provide complementation and feedback to solve the label confusion problem. By introducing contrastive learning technology, the present invention can better distinguish similar or confusing labels and improve the accuracy and reliability of classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the present invention will be briefly introduced below. It is obvious that the drawings described below are only some technical solution diagrams and embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A text representation module diagram of an intelligent document classification method based on contrast fusion perception is provided for the technical solution of the present invention.

[0074] Figure 2 A self-attention mechanism module diagram in an attention mechanism module of an intelligent document classification method based on contrast fusion perception is provided for the technical solution of the present invention.

[0075] Figure 3 A label semantic attention mechanism module diagram in an attention mechanism module of an intelligent document classification method based on contrast fusion perception is provided for the technical solution of the present invention.

[0076] Figure 4 A schematic diagram of a model in an intelligent document classification method based on contrast fusion perception provided by an embodiment of the present invention;

[0077] Figure 5A structural diagram of dynamic learning need perception in a learning need perception module in a deep knowledge point recommendation method based on learning need perception provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0079] In order to make the purpose, features and advantages of the present invention more obvious and understandable, the embodiments of the present invention are clearly and completely described below in conjunction with the specific implementation methods involved in the drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work, such as only changing the purpose without changing the embodiments based on the principles involved in the claims, are within the scope of protection of the present invention;

[0080] Embodiment 1: In order to solve the problems of inaccurate prediction and semantic confusion of labels in the existing document classification methods, resulting in low classification accuracy and poor use effect, refer to the attached Figure 4 Attached Figure 4 A model schematic diagram of an intelligent document classification method based on contrast fusion perception provided by an embodiment of the present invention. An embodiment of the present invention provides an intelligent document classification method based on contrast fusion perception, and the intelligent document classification method based on contrast fusion perception includes:

[0081] The text representation module, attention mechanism module, label prediction module and contrastive learning module work in tandem. The text representation module obtains the contextual semantic representation of the text, and then the attention module extracts important features in the text semantics. The prediction module combines the text representation obtained by the attention module for classification prediction, and the label confusion contrast module is used to optimize the performance of the model. These modules work together to complete the text classification task. Specifically:

[0082] The main task of the text representation module is to convert the input text into a machine-understandable vector representation so that the subsequent classification task can accurately classify the text. In the text representation module, the AlBERT pre-trained model is first used to learn the vector representation of sentences and texts. AlBERT is a pre-trained language model based on the Transformer architecture, which aims to learn context-dependent representations of sentences and texts. The training process of AlBERT is unsupervised learning on a large-scale text corpus. It uses the Masked Language Modeling (MLM) task to train the model. In MLM, some words in the input text are randomly masked, and then the model needs to predict these masked words. In this way, AlBERT is forced to learn contextual information to better predict the masked words. This enables AlBERT to capture semantic and contextual information in sentences and texts. To use AlBERT in the text representation module, you first need to load the pre-trained AlBERT model. Then, we feed the input official document data into the AlBERT model for encoding. The input text is first tokenized to divide the sentence or paragraph into smaller words or subword units. These units pass through an embedding layer to convert them into fixed-dimensional word vectors. Then, through multiple layers of Transformer encoders, each word in the text is encoded into a context-dependent vector representation. These vectors capture the semantic and contextual information of the word in the text. The purpose of the subsequent LSTM module is to extract richer semantic information in the sentence. LSTM is a variant of a recurrent neural network (RNN) that captures long-range dependencies and contextual information in sequences through a gating mechanism. Taking the output of AlBERT as input, the LSTM module can further process the text sequence and extract richer semantic information. The internal structure of LSTM includes an input gate, a forget gate, and an output gate, which control the flow and preservation of information by learning weights. This enables LSTM to remember and utilize previous contextual information to provide support for subsequent tasks.

[0083] The attention mechanism module, connected to the text representation module, uses the self-attention mechanism and the label semantic attention mechanism to extract the key information of the input text and enhance the discriminability of the representation according to the semantic relationship of the classification label. First, the attention mechanism module uses the self-attention mechanism to assign attention weights by calculating the similarity between the query, key, and value. For the input text sequence, each word is mapped to the representation of the query, key, and value. These representations can capture the semantic and positional information of the word. In the self-attention calculation, the query of each word is used to calculate the degree of association with other words, that is, the similarity between the query and the key. Commonly used similarity calculation methods include dot product or other measurement methods. The similarity calculation results are normalized to obtain the attention weight, which is used to weight the sum of the word values. The self-attention mechanism can assign a weight to each word, reflecting the importance of the word in the entire text. Words with higher weights will receive more attention in subsequent calculations. In this way, the module can automatically learn the importance of different parts of the text and focus on the most discriminative information. In addition to the self-attention mechanism, the attention mechanism module also introduces the label semantic attention mechanism to further enhance the discriminability of the representation. The semantic information of the label is used as an additional input and interacts with the representation of the text. This allows each word to be assigned a label-related attention weight based on the semantic relationship between the text content and the classification label. Specifically, the representation of the label is similar to the representation of the text, and the label semantic attention weight is obtained through normalization. These weights can enable the module to pay more attention to the words related to the classification label, thereby improving the classification performance. The label semantic attention mechanism enables the module to highlight the text information related to the label in a targeted manner according to the requirements of the classification task. By combining the self-attention mechanism and the label semantic attention mechanism, the attention mechanism module can better capture the key information of the input text and enhance the distinguishability of the representation based on the semantic relationship of the classification label. This comprehensive attention mechanism improves the system's ability to understand the text, thereby improving the performance and effectiveness of the intelligent document classification system.

[0084] The contrast fusion module compares the semantic relationships between different text fragments and fuses them into a comprehensive representation to improve the accuracy and effect of document classification. The contrast fusion module first receives the vector sequence output by the attention mechanism module. These vector sequences will be used to perform contrast and fusion operations between different text fragments. The contrast operation measures the semantic relationship between different text fragments by calculating the similarity between them. The module uses cosine similarity to calculate the similarity between vectors. By comparing the similarity between two text fragments, the module can determine the degree of their semantic association. The fusion operation fuses the representations of different text fragments into a comprehensive representation to capture the semantic information of the entire text. Among them, the attention mechanism plays an important role in the contrast fusion module. Through the attention mechanism, the module can automatically learn the importance of different fragments and focus on the most discriminative information. For example, the self-attention mechanism can be used to assign attention weights to each fragment to strengthen the focus on key information. In the attention mechanism, the representation of each fragment is used as a query, key, and value, and the attention weight is obtained by calculating the similarity between them. These weights are used to weight the sum of the values ​​of the fragments to generate the fused representation. In this way, the contrastive fusion module can automatically learn the importance of each fragment and fuse them into a comprehensive representation. The output of the contrastive fusion module is a comprehensive representation of the entire text, which contains the semantic relationship and important information between different fragments. This comprehensive representation will be sent to the subsequent label prediction module for further processing and analysis. Through the contrastive fusion module, the intelligent official document classification system based on label semantic contrastive fusion perception can better capture the semantic relationship and important information of official document text. It can automatically learn the importance of different fragments in the text and fuse them into a comprehensive representation. This contrastive fusion method improves the system's ability to understand the text, thereby improving the accuracy and effect of official document classification.

[0085] The label prediction module automatically predicts the label or category to which the text belongs by connecting the attention mechanism module and the contrast fusion module. The label prediction module first receives the text representation processed by the contrast fusion module. These representations contain the semantic relationship and important information of the text and have been fused into a comprehensive vector representation. Next, the label prediction module uses these vector representations to predict the label of the text. In traditional machine learning algorithms, the label prediction module uses the text representation as input features and then uses training data for model training. During the training process, the module establishes a classification model by learning the relationship between the text representation and the label. This classification model can be used to predict the label to which a new text belongs. The label prediction module adopts a neural network structure. Through the combination and training of multi-layer neural networks, it learns the complex mapping relationship between text representation and label, thereby realizing label prediction. In the label prediction module, model performance evaluation is an important link. By comparing the predicted results with the true labels, the model's accuracy, recall rate, F1 value and other indicators are calculated to evaluate the performance and effect of the model. During the training process, the model performance is improved by adjusting the model structure, optimizing hyperparameters, and increasing training data.

[0086] In summary, the text representation module, attention mechanism module, label prediction module and contrast fusion module of the intelligent document classification method based on contrast fusion perception each play an important role. They work together to extract the semantics, contextual information and key features of the text, and finally realize the automatic classification and label prediction of the document.

[0087] Embodiment 2 Based on the above embodiment 1, Figure 5 , attached Figure 5 A flow chart of an intelligent document classification method based on contrast fusion perception provided by an embodiment of the present invention. This embodiment provides an intelligent document classification method based on contrast fusion perception, and the intelligent document classification method based on contrast fusion perception includes the following steps:

[0088] Step 1: The text representation module initializes the input document number, source unit and title data through the pre-trained model, and then performs vector encoding to convert the text data into a machine-understandable vector representation. Then all the obtained initialized features are input into the attention mechanism module. The text representation module of this embodiment includes pre-training, word segmentation and word embedding as described in the above embodiment 1. The specific implementation of pre-training, word segmentation and word embedding will not be repeated here.

[0089] Step 2, the attention mechanism module adopts the self-attention mechanism and the label semantic attention mechanism by receiving the information of the text representation module to extract the key information of the input text, and strengthen the distinctiveness of the representation according to the semantic relationship of the classification label. First, the attention mechanism module uses the self-attention mechanism to assign attention weights by calculating the similarity between the query, key and value. The semantic information of the label is used as an additional input and interacts with the representation of the text. In this way, the attention weight related to the label can be assigned to each word according to the semantic relationship between the text content and the classification label. The attention mechanism module of this embodiment includes a self-attention mechanism module and a label semantic attention mechanism module as described in Example 1. The implementation of the specific self-attention mechanism module and the label semantic attention mechanism module will not be repeated here.

[0090] Step 3, the contrast fusion module compares the semantic relationships between different text fragments and fuses them into a comprehensive representation to improve the accuracy and effect of document classification. The contrast operation measures the semantic relationship between different text fragments by calculating the similarity between them. The fusion operation fuses the representations of different text fragments into a comprehensive representation to capture the semantic information of the entire text. Finally, the contrast loss generated by the module is fused with the model itself for subsequent prediction. The contrast fusion module of this embodiment is implemented based on contrast learning technology as described in the above embodiment 1, and the specific implementation is not repeated here.

[0091] Step 4: Label prediction module

[0092] The label prediction module first receives the text representation processed by the comparison fusion module, and after adaptive attention fusion, learns the complex mapping relationship between the text representation and the label through the combination and training of the multi-layer neural network, thereby realizing the prediction of the label. The label prediction module of this embodiment implements the fusion operation based on the deep learning classifier as described in the above embodiment 1, and the specific implementation is not repeated here.

[0093] The component combination features not described in detail in the specification belong to the content that is easily thought of by the known technology or easily determined and undisputed when implementing the present invention. The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions made without departing from the concept of the present invention still fall within the scope of protection of the present invention.

[0094] The present invention provides an intelligent document classification system based on label semantic contrast fusion perception, which includes: a text representation module, an attention mechanism module, a label prediction module and a contrast fusion module. The text representation module is used to map each word in a sentence to a high-dimensional vector space, and capture the contextual semantic relationship between words through BILSTM, so as to obtain the contextual semantic representation of the sentence. Such a representation can better capture the semantic features of the sentence and provide a basis for the subsequent classification process; the attention mechanism module includes a self-attention mechanism and a label semantic attention mechanism. The self-attention mechanism is used to calculate the importance of each word in the sentence, generate an attention weight, and then obtain a comprehensive contextual semantic representation of the sentence. The label semantic attention mechanism is used to calculate the importance of each word for each label and generate a label-related contextual semantic representation. Through the combination of these two attention mechanisms, the semantic association between the sentence itself and the label can be better captured, thereby improving the accuracy of classification; the contrast fusion module is used to analyze the difference between the contextual semantics of the sentence itself obtained by the self-attention mechanism and the label-related information obtained by the label semantic attention mechanism, and perform contrast learning complementation and feedback to provide information to improve the accuracy of model classification. The label prediction module adaptively weights the contextual semantic representation obtained by the self-attention mechanism and the contextual semantic representation of label-related information obtained by the label semantic attention mechanism, and simultaneously samples and compares the results of the fusion module input to obtain the final sentence feature representation, and inputs it into the classifier to output the probability of predicting each responsibility label. By setting an appropriate threshold, the final predicted responsibility label can be determined.

[0095] The above-mentioned intelligent document classification system based on label semantic comparison and fusion perception is shown in the attached Figure 1 As shown, the text representation module includes two sub-modules: pre-training and BILSTM. The former is to convert the text into a trainable vector representation, and the latter is to extract richer semantic information in the sentence.

[0096] The pre-training submodule uses the ALbert pre-training model trained based on a large-scale text corpus, which is responsible for mapping the input text sequence to the pre-trained word embedding space and generating a high-dimensional text representation containing semantic information. n}, n represents the number of words in the sentence, and the vector representation of the words in the document is obtained after the pre-training module: W′={w′1, w′2, ..., w′ n},w i Represents the i-th word in the sentence, which is expressed as follows:

[0097] w′1=Albert(w1)

[0098] The BILSTM submodule is responsible for forward and backward processing of words in the sentence, obtaining the hidden states of the sentence in both the forward and reverse directions, effectively processing the long-distance dependencies in the input text sequence, and capturing the contextual information in the text sequence. The vector representation W′ of the text is passed through the LSTM module, and two inputs are obtained through the forward and reverse sentence sequences respectively:

[0099]

[0100] in, and h i is its corresponding forward / reverse word representation. Thus, the entire document can be represented as:

[0101]

[0102] in, Represents text in two directions.

[0103] The above-mentioned intelligent document classification system based on label semantic contrast fusion perception, the attention mechanism module includes two sub-modules: self-attention mechanism and label semantic attention mechanism. The former is to obtain important features of the text content itself, and the latter is to obtain important features associated with each label.

[0104] Among them, as attached Figure 2 As shown, the self-attention mechanism module calculates the correlation between each word, and then performs weighted aggregation on each word according to the weight of the correlation, and finally obtains a more comprehensive contextual semantic representation of the sentence itself.

[0105] The specific steps of the self-attention mechanism are as follows:

[0106] First, the query, key, and value are calculated. For each element in the input sequence, a query vector, a key vector, and a value vector are calculated. These vectors are obtained by linearly transforming the learned weight matrix and the input elements. The input is a one-way sequence H = {h1, h2*…*h n}, the weight matrix is ​​W Q , W K , W V , then for each element h i ,have:

[0107] Q i =W Q ·h i

[0108] K i =W K ·h i

[0109] V t =W1·h t

[0110] Among them, Q i , K i , V i Represent the query, key and value vectors of the i-th element respectively.

[0111] Next, the attention score is calculated, for each pair of elements h i and h j , an attention score is calculated, indicating h i For h j The attention score is calculated by taking the dot product of the query vector and the key vector and dividing it by a scaling factor (given by the square root of the key vector dimension):

[0112]

[0113] where d k is the dimension of the key vector.

[0114] Then the attention weight is calculated, and the softmax function is used to convert the attention score into a value between 0 and 1 and the sum is 1, thereby obtaining the attention weight:

[0115] a ij =softmax(score(Q i , K j ))

[0116] Finally, calculate the output, multiply the value vector of each element by its corresponding attention weight, and then sum them up to get the final output:

[0117]

[0118] In the above formula, V j represents the value vector of each element, m i Represents the generated new context-related word vector, which contains not only the information of the current word, but also the information of its context. Finally, all the word vectors form the sentence vector M s It has contextual meaning.

[0119] As attached Figure 3 As shown in FIG. 1 , the label semantic attention mechanism mainly calculates the correlation between each word and each label in the sentence, and then performs weighted aggregation on each word according to the weight of the correlation, and finally obtains the relevant information of the entire sentence for the label.

[0120] The specific steps of the label semantic attention mechanism are as follows: first, obtain the label semantic vector representation:

[0121] For a set of label texts E, the text content of any label is represented by e = {w1, w2, ..., w p}, p is the length of the text label, which is usually between 1 and 3. In order to obtain the representation c of each label i , using the text content of the label as input, first obtain the vector representation of each word through the pre-trained model, and then use the word vector average function to calculate the vector representation of the label, as described below:

[0122]

[0123] All labels in the dataset are represented as a matrix C. Next, the word-label correlation is calculated. When predicting different labels, each word in the document plays a different important role. In order to capture the potential relationship between labels and words in the text, an attention mechanism based on label semantic information is designed to obtain the importance of each word. The weight of each word to the current label is obtained by calculating the matching score between the word in the document and each label:

[0124] a=CH T

[0125] Get the matching scores of all tags and words:

[0126]

[0127] Get the matching score of all tags for each word in the current document

[0128] Finally, the output is calculated. From the matching score, we can get the part of the document that each tag pays more attention to, and then combine the text vector representation to get the document representation corresponding to the tag:

[0129]

[0130] Here The features that each tag pays more attention to in the forward and reverse sentence representations of the document, and finally M l The document representation corresponding to all tags.

[0131] The above-mentioned intelligent document classification system based on label semantic contrast and fusion perception, the contrast and fusion module mainly compares the important features extracted by the content-based self-attention mechanism and the label-related features obtained by the label semantics-based attention mechanism, so as to promote the model to extract features that are more effective for classification.

[0132] First, the text representation vectors obtained by the self-attention mechanism and the label semantic attention mechanism are normalized by softmax:

[0133] y (l) =softmax(M (l) )

[0134] y (s) =softmax(N (s) )

[0135] Secondly, in order to make the prediction model fully utilize this information, a KL divergence method is used for comparison, which can be expressed as follows:

[0136] Loss(KL)=KL-divergence(y (y) ,y (l) )

[0137] Then, the model itself uses the cross entropy loss function to update the model. The formula is as follows:

[0138]

[0139] where y ij It is a binary value used to identify whether text i belongs to label j, where n represents the total number of texts and l represents the total number of labels.

[0140] Finally, add the two losses to get the final loss representation:

[0141] Loss = αLoss(KL) + L

[0142] Where α is the weight of label confusion comparison loss.

[0143] The above-mentioned intelligent document classification system based on label semantic contrast fusion perception, the label prediction model is divided into two sub-modules: adaptive attention fusion and classification. The former combines the features extracted by the self-attention mechanism and the label-based label semantic attention mechanism to obtain the final text representation, and the latter classifies the text to obtain the probability value of the text for each label.

[0144] The adaptive attention fusion part uses an adaptive method to fuse the features extracted by the content-based self-attention mechanism and the tag-based label semantic attention mechanism, and selects the more important part from the features extracted by the above two attentions to generate a vector representation that is more representative of the document. Specifically, two weight vectors are used to obtain the importance of the above two parts of the document representation, which can be obtained through a fully connected layer:

[0145] α=sigmoid(M(s) W1)

[0146] β=sigmoid(M (l) W2)

[0147] Among them, W1 and W2 are parameters, which represent the importance of self-attention mechanism and label semantic attention mechanism respectively. So add restrictions to these two pairs of parameters:

[0148] α j +β j =1

[0149] Get the final document representation;

[0150] In the classification module, after obtaining the final document representation, a multi-label text classifier is established through two fully connected layers. The predicted probability of each label can be obtained by the following formula:

[0151]

[0152] Among them, W3 and W4 are parameters, representing the weight matrix of the fully connected layer. The value output by sigmoid can be regarded as probability. Here, we get the probability of the responsibility label corresponding to each document. The responsibility labels exceeding the threshold are retained, and the corresponding departments and leaders are found through the responsibility-department leader table to obtain the final prediction result.

Claims

1. An intelligent document classification system based on label semantic comparison and fusion perception, characterized in that: The system comprises: The text representation module is used to map each word in the input sentence into a high-dimensional vector space and use BILSTM to capture the contextual semantic relationship between words to obtain the contextual semantic representation of the sentence; Attention mechanism module, including self-attention mechanism and label semantic attention mechanism. The self-attention mechanism is used to calculate the importance of each word in a sentence and generate attention weights to obtain a comprehensive contextual semantic representation of the sentence. The label semantic attention mechanism is used to calculate the importance of each word for each responsibility label and generate contextual semantic representations related to the label. The contrast fusion module is used to analyze the difference between the comprehensive contextual semantic representation of the sentence obtained by the self-attention mechanism and the contextual semantic representation related to the label obtained by the label semantic attention mechanism to obtain the Loss; The difference between the contextual semantics of the sentence itself obtained by the self-attention mechanism and the label-related information obtained by the label semantic attention mechanism is compared and learned, and feedback is given to obtain the contrast loss. The weights are then added through the cross-entropy loss function to obtain the difference between the probability distribution of the model prediction and the probability distribution of the true label, which is used as a basis for improving the classification accuracy of the model. The label prediction module is used to adaptively weight the comprehensive contextual semantic representation of the sentence obtained by the self-attention mechanism and the contextual semantic representation related to the label obtained by the label semantic attention mechanism, and combine the model prediction probability distribution of the contrast fusion module and the probability distribution difference of the true label to obtain the final sentence feature representation. The final sentence feature representation is input into the classifier to output the probability of predicting each responsibility label, and the final predicted responsibility label can be determined by setting an appropriate threshold.

2. According to claim 1, the intelligent document classification system based on label semantic comparison and fusion perception is characterized in that: The text representation module includes two submodules: pre-training and BILSTM. The pre-training submodule is used to convert the text into a trainable vector representation, and the BILSTM submodule is used to extract semantic information in the sentence. The pre-training submodule uses the ALbert pre-training model trained based on a large-scale text corpus to map the input text sequence to the pre-trained word embedding space to generate a high-dimensional text representation containing semantic information; Given a sentence in a document, its word sequence W = {w1, w2, ..., w n }, n represents the number of words in the sentence, and the vector representation of the words in the document is obtained after the pre-training module: W′={w′1,w′2,...,w′ n },w i Represents the i-th word in the sentence, which is expressed as follows: w′ i =Albert(w i ) The BILSTM submodule is used to process the words in the sentence forward and backward, obtain the hidden states of the sentence in both the forward and reverse directions, and obtain the context information in the text sequence; The vector representation W′ of the text passes through the LSTM module and gets two inputs through the forward and reverse sentence sequences respectively: in, and h i is its corresponding forward / reverse word representation, so the entire document can be represented as: in, Represents text in two directions.

3. The intelligent document classification system based on label semantic comparison and fusion perception according to claim 2 is characterized in that: The specific steps of the self-attention mechanism are as follows: First, the query, key, and value are calculated. For each element in the input sequence, a query vector, a key vector, and a value vector are calculated. These vectors are obtained by linearly transforming the learned weight matrix and the input elements. The input is a unidirectional sequence H = {h1, h2, ..., h n }, the weight matrix is ​​W Q , W K , W V , then for each element h i ,have: Q i =W Q ·h t K t =W k ·h t V t =W V ·h i Among them, Q i , K i , V i Represent the query, key, and value vectors of the i-th element respectively; Next, the attention score is calculated, for each pair of elements h i and h j , an attention score is calculated, indicating h i For h j The attention score is calculated by taking the dot product of the query vector and the key vector and then dividing it by a scaling factor: where d k is the dimension of the key vector; Then the attention weight is calculated, and the softmax function is used to convert the attention score into a value between 0 and 1 and the sum is 1, thereby obtaining the attention weight: a ij =softmax(score(Q i ,K j )) Finally, calculate the output, multiply the value vector of each element by its corresponding attention weight, and then sum them up to get the final output: In the above formula, V j represents the value vector of each element, m i Represents the generated new context-related word vector, which contains the information of the current word and its context. Finally, all the word vectors form the sentence vector M s Has contextual meaning.

4. The intelligent document classification system based on label semantic comparison and fusion perception according to claim 3 is characterized in that: The specific steps of the label semantic attention mechanism are as follows: first, obtain the label semantic vector representation: For a label text set E, the text content of any label is represented by e = {w1, w2, ..., w p } It is the length of the text label. The length of the text label is generally between 1 and 3. In order to obtain the representation c of each label i , using the text content of the label as input, first obtain the vector representation of each word through the pre-trained model, and then use the word vector average function to calculate the vector representation of the label, as described below: All the labels in the dataset are represented as a matrix C. Next, the word-label correlation is calculated. An attention mechanism based on label semantic information is designed to obtain the importance of each word. The weight of each word to the current label is obtained by calculating the matching score between the word in the document and each label: a=CII T Get the matching scores of all tags and words: Get the matching score of all tags for each word in the current document Finally, the output is calculated. From the matching score, we can get the part of the document that each tag pays more attention to, and then combine the text vector representation to get the document representation corresponding to the tag: Here It is the feature that each label pays more attention to in the forward and reverse sentence representation of the document, and finally M is obtained. l The document representation corresponding to all tags.

5. The intelligent document classification system based on label semantic comparison and fusion perception according to claim 4 is characterized in that: The comparison and fusion module has the following specific steps: First, the text representation vectors obtained by the self-attention mechanism and the label semantic attention mechanism are normalized by softmax: and (l) =softmax(M (l) ) and (s) =softmax(M (s) ) Secondly, in order to make the prediction model fully utilize this information, a KL divergence method is used for comparison, which can be expressed as follows: Loss(KL)=KL-divergence(y (s) ,y (l) ) Then, the model itself uses the cross entropy loss function to update the model. The formula is as follows: Where yij is a binary value used to identify whether text i belongs to label j, where n represents the total number of words and l represents the total number of labels; Finally, add the two losses to get the final Loss representation: Loss = αLoss(KL) + L Where α is the weight of label confusion comparison loss.

6. The intelligent document classification system based on label semantic comparison and fusion perception according to claim 5 is characterized in that: The label prediction module is divided into two sub-modules: adaptive attention fusion and classification. The adaptive attention fusion sub-module combines the features extracted by the self-attention mechanism and the label-based label semantic attention mechanism to obtain the final text representation. The classification sub-module classifies the text to obtain the probability value of the text for each label.

7. The intelligent document classification system based on label semantic contrast and fusion perception according to claim 6 is characterized in that: The adaptive attention fusion part uses an adaptive method to fuse the features extracted by the content-based self-attention mechanism and the tag-based label semantic attention mechanism, and selects the more important part from the features extracted by the above two attentions to generate a vector representation that is more representative of the document. Specifically, two weight vectors are used to obtain the importance of the above two parts of the document representation, which can be obtained through a fully connected layer: α=sigmoid(M (s) W1) β=sigmoid(M (l) W2) Among them, W1 and W2 are parameters, which represent the importance of self-attention mechanism and label semantic attention mechanism respectively. So add restrictions to these two pairs of parameters: α j +β j ＝1 Get the final document representation: M=αM (S) +βM (l) 8. The intelligent document classification system based on label semantic comparison and fusion perception according to claim 6 is characterized in that: In the classification submodule, after obtaining the final document representation, a multi-label text classifier is established through two fully connected layers. The predicted probability of each label can be obtained by the following formula: Among them, W3 and W4 are parameters, representing the weight matrix of the fully connected layer. The value output by sigmoid can be regarded as probability. What is obtained here is the probability of the responsibility label corresponding to each document. The responsibility labels exceeding the threshold are retained, and the corresponding departments and leaders are found through the responsibility-department leader table to obtain the final prediction result.

Images