Document classification method and electronic equipment

Abstract

The invention discloses a document classification method and electronic equipment, and relates to the technical field of computers.The document classification method comprises the steps that multi-dimensional information of a to-be-classified document is extracted from multiple dimensions, multi-dimensional feature vectors are extracted through different feature extraction methods according to different dimension information, and the multi-dimensional feature vectors are used for classifying the to-be-classified document; a weighted fusion method based on an attention mechanism is adopted to construct comprehensive document feature vectors, and meanwhile, importance weights of feature vectors of all dimensions are dynamically adjusted, so that features which contribute to classification greatly obtain higher weights, the effectiveness of feature fusion is improved, and the accuracy of feature fusion is improved. And finally, based on the pre-trained classification model, realizing accurate classification based on the document feature vector, thereby effectively solving the technical problem that the document classification accuracy and robustness are seriously insufficient, and achieving the technical effect of improving the document classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a document classification method and electronic equipment. BACKGROUND

[0002] With the rapid development of information technology, the number of documents in various knowledge bases presents exponential growth, covering various types such as academic, technical, business, news, etc. How to efficiently and accurately classify them so that users can quickly search and use them has become a key problem in knowledge base management.

[0003] At present, the related document classification methods often rely on manual annotation or keyword matching. The former has high cost and strong subjectivity, and is difficult to meet the large-scale document classification demand. The latter cannot deeply understand the semantic connotation of the document, and is difficult to meet the semantic diversity and vocabulary variation. It is easy to make classification errors for documents with similar semantics but different vocabularies, and has poor adaptability. With the development of natural language processing and deep learning technology, the document classification method based on semantic understanding has gradually become a research hotspot. However, most of the related document classification methods are based on single-dimensional semantic feature extraction, which leads to one-sided document feature vector representation and incomplete semantic expression, resulting in serious lack of classification accuracy and robustness. SUMMARY

[0004] The present application provides a document classification method and electronic equipment to at least solve the problem of serious lack of classification accuracy and robustness in the related art.

[0005] The present application provides a document classification method, comprising: extracting information from multiple dimensions of a document to be classified to obtain multiple document information; extracting features from the multiple document information to obtain multiple dimension feature vectors, wherein each document information corresponds to a dimension feature vector; calculating the importance weight of the multiple dimension feature vectors according to the activation function and the set attention weight parameter, and performing weighted fusion on the multiple dimension feature vectors and the corresponding importance weight to obtain a document feature vector; classifying the document based on the document feature vector through a pre-trained classification model to obtain a document category of the document to be classified.

[0006] The present application also provides a document classification device, comprising: an information extraction unit configured to extract information from multiple dimensions of a document to be classified to obtain multiple document information; a feature extraction unit configured to extract features from the multiple document information to obtain multiple dimension feature vectors, wherein each document information corresponds to a dimension feature vector; The computing unit is configured to calculate the importance weight of the plurality of dimensional feature vectors according to the activation function and the set attention weight parameter, and to obtain a document feature vector by weighted fusion of the plurality of dimensional feature vectors and the corresponding importance weight. The classification unit is configured to perform document classification on the document feature vector by using a pre-trained classification model, and to obtain a document category of the document to be classified.

[0007] The present application also provides an electronic device, comprising a memory for storing a computer program, and a processor for executing the computer program to implement the steps of any of the above document classification methods.

[0008] The present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the above document classification methods.

[0009] The present application also provides a computer program product, which comprises a computer program, wherein the computer program is executed by a processor to implement the steps of any of the above document classification methods.

[0010] The document classification method provided by the present application extracts multi-dimensional information of the document to be classified from multiple dimensions, extracts multi-dimensional feature vectors by different feature extraction methods for different dimensional information, adopts a weighted fusion method based on an attention mechanism to construct a comprehensive document feature vector, dynamically adjusts the importance weight of each dimensional feature vector, so that the features with greater contribution to classification obtain higher weight, improves the effectiveness of feature fusion, and finally realizes accurate classification based on the document feature vector based on a pre-trained classification model. Therefore, the present application can solve the technical problems of serious lack of document classification accuracy and robustness, and achieve the technical effect of improving the accuracy of document classification. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 A flowchart of a document classification method provided by an embodiment of the present application; Figure 2 A flowchart of another document classification method provided by an embodiment of the present application; Figure 3 A structural diagram of a document classification system provided by an embodiment of the present application; Figure 4Another structural diagram of a document classification system provided by an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0014] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0016] In combination with the specific application environment architecture or specific hardware architecture on which the execution of the document classification method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0017] Before the present application is explained, the professional data involved is explained first, including: NLP (Natural Language Processing): a general term for technologies that enable computers to understand, generate, and process human language.

[0018] BERT (Bidirectional Encoder Representations from Transformers): a pre-trained language model that excels in semantic understanding and supports fine-tuning for various NLP tasks.

[0019] LSTM (Long Short-Term Memory): excels in processing long sequence dependencies and is commonly used for text modeling and time series prediction.

[0020] CNN (Convolutional Neural Network): Used for local feature extraction in text, such as keywords and phrase patterns.

[0021] TF-IDF (Term Frequency-Inverse Document Frequency): A statistical method to measure the importance of words in documents, commonly used for keyword extraction and text vectorization.

[0022] Word2Vec (Word Embedding Model): Used to map words to dense vectors, capturing semantic similarity.

[0023] KB (Knowledge Base): A structured knowledge system that stores entities, concepts, and their relationships, used for knowledge reasoning and question answering.

[0024] SVM (Support Vector Machine): A traditional machine learning classifier, good at high-dimensional space classification, widely used in text classification.

[0025] MLP (Multi-Layer Perceptron): A feedforward neural network, which can be used for classification, regression, and other tasks.

[0026] In today's information explosion era, the number of documents in various knowledge bases is growing exponentially, covering academic papers, technical manuals, corporate reports, news information, and other types. These documents carry a wealth of knowledge and information, and how to efficiently and accurately classify them so that users can quickly search and use them has become a key problem in knowledge base management.

[0027] Related document classification methods often rely on manual annotation or simple keyword matching, which have many limitations. For example, manual annotation is time-consuming and labor-intensive, and is greatly affected by subjective factors, making it difficult to meet large-scale document classification needs. Keyword matching methods can classify based on surface lexical information, but they cannot deeply understand the deep semantics of documents, and are prone to classification errors for documents with similar semantics but different keywords, and have poor adaptability. For example, for documents in the "computer network" category, the preset keywords include "router" and "switch". When the frequency of these keywords in the document to be classified reaches a certain threshold, it will be classified into this category. However, for "network equipment" and "router" and "switch" keywords that are semantically related, this method cannot identify this association, resulting in classification errors. At the same time, this method lacks consideration of the context information and semantic structure of the document, and the classification accuracy is low, making it difficult to adapt to complex knowledge base document classification scenarios.

[0028] With the development of natural language processing and deep learning technologies, document classification methods based on semantic understanding have gradually become a research hotspot. However, existing semantic-based classification methods mostly stop at single-dimensional semantic feature extraction and fail to fully integrate multi-dimensional information of documents, leaving room for improvement in classification accuracy and robustness. For example, after converting the words in the document into word vectors, the document vector is obtained by averaging, and then input into the classifier for category prediction. This method can capture some semantic information, but it is based on semantic features at the word level and fails to integrate other dimensional features of the document. Moreover, the word vector model has limited ability to handle complex semantic phenomena such as polysemy, resulting in insufficient accuracy of document vector representation and affecting the classification effect.

[0029] Embodiments of the present application provide a document classification method, aiming to realize more accurate and efficient intelligent classification of knowledge base documents. The method is described in detail through the following embodiments.

[0030] Figure 1 A flowchart of a document classification method provided by an embodiment of the present application is shown in detail, which includes the following steps as shown in Figure 1 S101, information extraction of the document to be classified from multiple dimensions to obtain multiple document information.

[0031] It can be understood that the document to be classified can be obtained from various knowledge bases such as academic databases, enterprise knowledge bases, and network information platforms. The format of the target document can be a text file (such as txt, docx) or a web page file (such as html). In addition, the content of the target document can cover the target field and the target topic. Then, information extraction is performed on the target document from multiple dimensions to obtain multiple document information, i.e., each dimension corresponds to a document information, and multiple document information can also be understood as multi-dimensional information. Among them, the multiple dimensions include at least two dimensions of semantic, structure, field, sentiment, and time sequence, and other possible dimensions are not limited.

[0032] Optionally, the information extraction of the document to be classified from multiple dimensions to obtain multiple document information includes: standardizing the document to be classified to obtain text content information, wherein the standardization processing includes word segmentation processing; parsing the format of the document to be classified, determining at least one logical structure, and recording the position and hierarchical relationship of at least one logical structure in the document to be classified to obtain document structure information; extracting the attribute information of the document to be classified to obtain document attribute information; wherein the multiple document information includes at least one content information in the text content information, the document structure information, and the document attribute information.

[0033] ​It can be understood that after obtaining the target document, a preprocessing operation is performed on the target document, and the preprocessing operation includes data cleaning, text standardization, structure information extraction, and metadata extraction. Specifically, the data cleaning operation refers to removing noise data such as garbled codes, special symbols, and repeated content in the target document. In addition, for the scanned target document, it can be converted into editable text through OCR technology. The text standardization operation refers to converting English text in the target document into lowercase English; performing morphological reduction on the English text (for example, reducing the English verb + tense to the original English); performing word segmentation processing on the Chinese text (for example, using the jieba word segmentation tool); removing stop words (such as “of”, “is”, “in” and other meaningless words), and finally obtaining text content information. The structure information extraction operation refers to parsing the format of the target document, determining at least one logical structure, such as title, abstract, chapter title, paragraph, and other structure information, and recording the position and hierarchical relationship of each part in the target document to obtain document structure information. The metadata extraction operation refers to extracting metadata such as author, publication time, keywords, and field of the target document from the attribute information or header and footer of the target document to obtain document attribute information. Among them, the plurality of document information includes at least one content information in the text content information, the document structure information and the document attribute information, for example, the plurality of document information includes the text content information, the document structure information and the document attribute information, and for example, the plurality of document information includes the text content information and the document attribute information. Subsequently, the preprocessed plurality of document information can also be stored in a certain format to form a structured document dataset, preparing for subsequent feature extraction.

[0034] S102, performing feature extraction on the plurality of document information to obtain a plurality of dimension feature vectors.

[0035] Among them, each document information corresponds to a dimension feature vector, and the plurality of dimension feature vectors includes at least one feature vector extracted from the text content information, the structure feature vector extracted from the document structure information, and the field feature vector extracted from the document attribute information.

[0036] It can be understood that on the basis of the above S101, feature extraction is performed on each document information respectively to obtain the dimension feature vector corresponding to each document information, that is, the plurality of document information corresponds to a plurality of dimension feature vectors. For example, the plurality of dimension feature vectors include semantic feature vectors extracted from the text content information, structure feature vectors extracted from the document structure information, and field feature vectors extracted from the document attribute information. Different dimensions of feature extraction are described in the following embodiments and will not be described here.

[0037] Optionally, performing feature extraction on the plurality of document information to obtain a plurality of dimension feature vectors, comprising: The text content information is semantically encoded to obtain deep semantics, and a semantic feature vector is obtained; a structural feature vector is extracted based on document structure information, wherein the structural feature vector is used to represent the organization mode and importance of the document; and a domain feature vector is extracted from document attribute information through a pre-constructed domain dictionary and a pre-trained topic model; wherein the multiple dimension feature vectors include at least one of the semantic feature vector, the structural feature vector and the domain feature vector.

[0038] It can be understood that the text content of the target document is semantically encoded by using the pre-trained BERT model to obtain deep semantic features, and a semantic feature vector is obtained. Based on the structure information of the target document, structural features are extracted to obtain a structural feature vector, wherein the structural feature vector is used to reflect the organization mode and importance of the document. In combination with the domain dictionary and the LDA (Latent Dirichlet Allocation, topic model), the domain feature vector of the document is extracted based on the attribute information of the target document. This kind of multi-dimensional feature extraction method can deeply mine the deep semantic association of the document, reflect the organization architecture and importance of the document, and embody the professional characteristics of the domain to which the document belongs, effectively improving the expression ability and discrimination of the features, and laying a foundation for accurate classification of the classification model.

[0039] Optionally, the semantic encoding of the text content information to obtain deep semantics to obtain a semantic feature vector comprises: The text content information is segmented and encoded according to the input requirements of the pre-trained language model to generate a word embedding vector for each word, wherein the text content information includes a plurality of words; a context-related vector representation of each word is extracted based on the word embedding vector of each word by the language model to obtain a word vector of each word; a first attention weight between different words is calculated based on the word vector of each word, wherein the attention weight is used to represent the weight of different words in semantic association; and the word vector is weighted and summed according to the first attention weight to obtain a semantic feature vector.

[0040] As can be understood, the preprocessed text content is segmented and encoded according to the input requirements of the pre-trained language model (such as the BERT model) to generate word embedding vectors, wherein after the text content is segmented, a plurality of words are obtained, that is, the text content information includes a plurality of words, and the word embedding vectors of each word can be obtained through the BERT model. Subsequently, the word embedding vectors are input into the BERT model, and feature extraction is performed through a multi-layer Transformer encoder to obtain a context-related vector representation of each word, that is, a word vector of each word. A bidirectional attention mechanism is introduced, and the semantic correlation weight between different words in the document is calculated according to the word vector of each word, denoted as the first attention weight, which is used to measure the numerical value of the semantic correlation between two words, and the higher the numerical value, the more relevant the two words are in meaning, context, function or knowledge (that is, the higher the numerical value, the stronger the correlation), for example, the attention weight between word a and different words (such as word b and word c) is calculated, which is a quantitative value of semantic similarity or correlation strength, usually in the interval [0, 1] or [-1, 1]. Subsequently, for each word, the word vectors are weighted and summed according to the first attention weight to obtain a semantic feature vector of the entire target document. Through the semantic depth perception technology and the bidirectional attention mechanism, the deep semantic of the document is accurately captured, and the semantic, structural and domain features are fully integrated subsequently to improve the classification accuracy.

[0041] Optionally, the plurality of words includes a first word and a second word, and the first attention weight between different words is calculated based on the word vector of each word, including: The product of the word vector of the first word, the word vector of the second word and the first set weight matrix is calculated to obtain the correlation score between the first word and the second word; the sum of the correlation scores between the first word and other words in the plurality of words except the first word is calculated, wherein the other words include the second word; and the ratio of the correlation score and the sum is calculated to obtain the first attention weight between the first word and the second word.

[0042] As can be understood, the following embodiments are described in detail taking the calculation of the first attention weight between the first word (such as word a) and the second word (such as word b) as an example. The product of the word vector of the first word, the word vector of the second word and the first set weight matrix is calculated to obtain the correlation score between the first word and the second word, as shown in formula (1). Subsequently, the sum of the correlation scores between the first word and other words (such as word b and word c) is calculated, and the other words include the second word and may also include the first word itself. The ratio of the correlation score between the first word and the second word and the sum is calculated to obtain the first attention weight, as shown in formula (2). The attention weight calculation between other words is not described in detail.

[0043] Formula (1) wherein the word vector in the document is, is the number of words, is the association score between the first word and the second word, is the word vector of the second word, is the word vector of the first word, W is a first set weight matrix, and W is a learnable weight matrix.

[0044] Formula (2) wherein, is the first attention weight, is the sum value of the association scores between the first word and other words.

[0045] Optionally, the word vectors are weighted and summed according to the first attention weights to obtain a semantic feature vector, including: For each word, the word vector of each word is weighted and summed according to the attention weights between each word and other words in the plurality of words except each word to obtain a sentence-level sentence vector; the second attention weights between different sentence vectors are calculated; and the sentence vectors are weighted and summed according to the second attention weights to obtain a document-level semantic feature vector.

[0046] Understandably, the calculation process of the semantic feature vector of the entire document is as follows: for each word, the word vector is weighted and summed according to the attention weights between the word and other words to obtain a sentence-level sentence vector. Subsequently, the same attention weight calculation (i.e., calculating the semantic association weights between two sentence vectors) and weighted sum processing are performed on the sentence vectors to obtain a document-level semantic feature vector, denoted as S, wherein d is the feature dimension.

[0047] Optionally, the document structure information includes at least one of document title information, document abstract information, chapter title information, and chapter paragraph information, and a structure feature vector is extracted based on the document structure information, including: The document title information is encoded into a title feature vector by a pre-trained language model; the document abstract information is encoded into an abstract feature vector by a language model; the chapter title information and the chapter paragraph information are respectively encoded by a language model, and the structure features corresponding to the chapter title information and the chapter paragraph information are fused by an attention mechanism to obtain a chapter vector; the chapter feature vector is obtained by weighted summing all chapter vectors according to a set chapter weight, wherein the chapter weight is set according to the level of the chapter in the document; and the structure feature vector is obtained based on the title feature vector, the abstract feature vector, and the chapter feature vector.

[0048] It can be understood that the document title information is encoded into a title feature vector by a BERT model, denoted as The document abstract information is encoded into an abstract feature vector by a BERT model, denoted as The chapter title information and the chapter content / paragraph information of each chapter are encoded respectively to obtain a first encoding vector corresponding to the chapter title information and a second encoding vector corresponding to the chapter paragraph information, and the first encoding vector and the second encoding vector are fused by an attention mechanism to obtain a chapter vector. The chapter vectors are weighted and summed to obtain a chapter feature vector, denoted as wherein the weight can be determined according to the level of the chapter in the document, and the higher the level, the greater the weight, for example, the weight of the 1.1 chapter is 0.6, and the weight of the 1.1.1 chapter under the 1.1 chapter is 0.3. Subsequently, the title feature vector, the abstract feature vector and the chapter feature vector are spliced to obtain a structural feature vector, denoted as This kind of way has excellent processing ability for complex semantics such as polysemy and metaphor, and can mine key features from title, chapter and other structural information.

[0049] Optionally, a domain feature vector is extracted from the document attribute information by a pre-constructed domain dictionary and a pre-trained topic model, including: At least one domain dictionary is obtained, wherein the domain dictionary includes at least one core term corresponding to the domain; the frequency of occurrence of at least one core term included in each domain dictionary in the document attribute information is calculated to obtain a domain term frequency vector; the probability distribution of the target document on at least one set topic corresponding to the topic model is calculated based on the document attribute information by a pre-trained topic model to obtain a topic feature vector; and the domain feature vector is obtained according to the domain term frequency vector and the topic feature vector.

[0050] It can be understood that a plurality of professional dictionaries of multiple domains (also referred to as domain dictionaries) are obtained, each professional dictionary containing a plurality of core terms of the domain, for example, computer domain, medical domain, financial domain, etc. After selecting a target domain dictionary from the multiple domain dictionaries based on the document attribute information, the frequency of occurrence of the plurality of core terms included in the target domain dictionary in the target document is calculated to obtain a domain term frequency vector, or the frequency of occurrence of the plurality of core terms included in each domain dictionary in the target document is directly calculated to obtain a domain term frequency vector, denoted as wherein m is the number of domains. Subsequently, the probability distribution of the target document on each topic is calculated by using a topic model to obtain a topic feature vector, denoted as where k is the number of topics. It can be understood that the topic model is trained based on the document set, and the topic distribution of the preset number can be calculated. The domain term frequency vector and the topic feature vector are spliced to obtain a domain feature vector, denoted as .

[0051] S103, according to the activation function and the set attention weight parameter, the importance weight of the plurality of dimension feature vectors is calculated, and the plurality of dimension feature vectors and the corresponding importance weight are weighted and fused to obtain the document feature vector.

[0052] It can be understood that, on the basis of the above S102, the weighted fusion method based on the attention mechanism is used to fuse the semantic feature vector, the structural feature vector and the domain feature vector into a comprehensive document feature vector. For details, see the following embodiments.

[0053] Optionally, according to the activation function and the set attention weight parameter, the importance weight of the plurality of dimension feature vectors is calculated, and the plurality of dimension feature vectors and the corresponding importance weight are weighted and fused to obtain the document feature vector, including: According to the second set weight matrix and the set bias term, the linear transformation is performed on each dimension feature vector to unify the dimension feature vectors; according to the activation function and the set attention weight parameter, the importance weight of the linearly transformed dimension feature vectors is calculated; the importance weight of the linearly transformed dimension feature vectors is normalized to obtain the normalized weight of the linearly transformed dimension feature vectors; according to the normalized weight of the linearly transformed dimension feature vectors and the linearly transformed dimension feature vectors, the fused document feature vector is calculated.

[0054] It can be understood that, according to the second set weight matrix and the set bias term, the linear transformation is performed on each dimension feature vector (such as the above three dimension feature vectors) to obtain the feature vector of the unified dimension, and the specific calculation is shown in formula (3). Subsequently, according to the activation function and the set attention weight parameter, the importance weight of the linearly transformed dimension feature vectors is calculated, and the specific calculation is shown in formula (4). The importance weight of the linearly transformed dimension feature vectors is normalized to obtain the normalized weight of the linearly transformed dimension feature vectors, and the specific calculation is shown in formula (5). Subsequently, according to the normalized weight of the linearly transformed dimension feature vectors and the linearly transformed dimension feature vectors, the fused document feature vector is calculated, and the specific calculation is shown in formula (6).

[0055] Formula (3) In the formula, is the second weight matrix, To set the bias term, the semantic feature vector is S, the structure feature vector is St, and the domain feature vector is D.

[0056] Equation (4) In the formula, sigma is a sigmoid activation function, and is an attention weight parameter.

[0057] Equation (5) In the formula, , and is a normalization weight.

[0058] Equation (6) In the formula, V is a document feature vector.

[0059] S104, classifying the document based on the document feature vector through the pre-trained classification model, to obtain a document category of the document to be classified.

[0060] Understandably, on the basis of the above S103, the document feature vector of the target document is input into the trained classification model, and the model outputs the probability distribution of the target document belonging to each category, that is, , and the document category with the largest probability is taken as the final classification result of the target document, that is, , which can quickly and accurately classify the knowledge base document and improve the classification efficiency. The specific classification model is not limited.

[0061] Understandably, the classification result of the target document is evaluated, and the classification model and the above-mentioned weight parameters, weight matrices, etc. are optimized according to the evaluation result.

[0062] Optionally, the training process of the classification model is as follows: A training data set is constructed, wherein the training data set includes document category labels of document samples and fusion feature vectors of the document samples, wherein the fusion feature vector is obtained by weighting and fusing the multi-dimensional features of the document samples; the parameters of the classification model are initialized, and the fusion feature vector is taken as the input of the classification model, and the document predicted category of the document sample is output; a weighted cross-entropy loss function is used to calculate the loss value based on the document predicted category and the document category label, and the model parameters of the classification model are updated according to the loss value.

[0063] Understandably, document data is collected from various knowledge bases (servers) by calling the API (Application Programming Interface) interface provided by the knowledge base or using crawler technology, where the various knowledge bases include academic databases, enterprise knowledge bases, and network information platforms. The collected document data is used as a document sample, and the categories of the document sample are labeled to construct a training data set, where the training data set includes a large number of training samples, each training sample includes a fusion feature vector of the document sample and a corresponding document category label (such as a one-hot encoded category label). The training data set is divided into a training set and a validation set, for example, the division ratio is 8:2. Then, the parameters of the classification model are initialized, and the learning rate, batch size, number of iterations, and other hyperparameters are set. The cross-entropy loss function is used as the optimization objective of the classification model, and the loss function is shown in formula (7). At the same time, the model parameters of the classification model are updated using the Adam optimizer, and the loss function is minimized through the backpropagation algorithm to obtain a trained classification model that can accurately identify multiple document categories and has good generalization ability, ensuring adaptability and reliability in complex document classification scenarios in practical applications.

[0064] Formula (7) In the formula, L is the loss function, c is the number of categories, is the i-th component of the true label, that is, the document category label of the i-th document sample, is the i-th component of the classification model prediction probability.

[0065] Understandably, the model training adopts a dynamic optimization strategy to balance efficiency and performance, and can quickly adapt to documents in different fields and is stable in the face of data noise and format changes. At the same time, it has excellent scalability and can easily add feature dimensions or adapt to new fields. In the small sample scenario, it still maintains high accuracy through incremental learning. Compared with the "black box" model, it has stronger interpretability and can clearly show the contribution of each dimension feature to the classification result, enhancing user trust and providing an efficient, accurate and reliable solution for knowledge base document classification.

[0066] Optionally, the training data set includes a training set and a validation set, the training set is used to train the classification model, and the validation set is used to verify the classification model trained based on the training set. In the process of verifying the classification model based on the validation set, the method further comprises: Calculate the loss value of multiple training batches; if the loss value of a continuous preset number of training batches decreases by less than a preset threshold, stop training the classification model.

[0067] Understandably, in the training process of the classification model, an early stopping strategy is adopted, when the loss value on the validation set does not decrease or the decrease amplitude is less than a preset threshold for 5 (preset number) consecutive epochs (training batches), that is, the loss is not reduced for consecutive multiple times of training, the training is stopped to prevent overfitting of the model.

[0068] Understandably, in order to solve the problem of sample class imbalance during model training, the training samples are weighted, and higher weights are given to the classes with fewer samples, as shown in formula (8). Correspondingly, the weighted cross-entropy loss function is shown in formula (9).

[0069] Formula (8) In the formula, is the number of the i-th class sample, and the weight of the training sample of this class.

[0070] Formula (9) In the formula, is the weighted cross-entropy loss function, which can be used as the optimization target of the classification model. The loss value is calculated according to the document prediction category and the document category label, and the model parameters of the classification model are updated through the loss value. The specific parameter updating method is not described.

[0071] Optionally, after updating the model parameters of the classification model according to the loss value, the method further comprises: According to the evaluation results, the reasons for the classification error of the classification model for the target document are analyzed, and the analysis results are obtained. Based on the analysis results, the number of training samples of the target document is increased, the feature weight of the target category document is adjusted, and / or the incremental learning strategy is used to fine-tune the classification model, so as to continuously improve the classification accuracy and enhance the adaptability and stability of the system.

[0072] Understandably, the evaluation indicators include at least one of accuracy, precision, recall and F1 value. Among them, the accuracy is the ratio of the number of correctly classified documents to the total number of documents, as shown in formula (10). The precision is the ratio of the number of correctly classified documents in a certain category to the number of documents predicted as this category, as shown in formula (11). The recall is the ratio of the number of correctly classified documents in a certain category to the actual number of documents in this category, as shown in formula (12). The F1 value is the harmonic mean of the precision and the recall, as shown in formula (13).

[0073] Formula (10) In the formula, For accuracy, TP is true positive, i.e. actually positive class, model also predicts positive class, TN is true negative, i.e. actually negative class, model also predicts negative class, FP is false positive, i.e. actually negative class, model misjudges as positive class, FN is false negative, i.e. actually positive class, model misjudges as negative class.

[0074] Equation (11) In the formula, is the precision rate.

[0075] Equation (12) In the formula, is the recall rate.

[0076] Equation (13) Understandably, the accuracy, precision, recall and F1 value are used as the evaluation index of the classification result, and the evaluation result is generated. Subsequently, the analysis result is obtained by analyzing the reason for the classification error of the target document according to the evaluation result. If the classification effect of a certain document category is poor, it may be due to insufficient training samples of the category or insufficient feature extraction. For this kind of situation, the number of training samples of the category can be increased, for example, the samples are expanded through data enhancement technology (such as synonym replacement, sentence rearrangement, etc.). The feature weight related to the category can also be adjusted to enhance the contribution of the features that have distinguishing characteristics for the category. Incremental learning strategy can also be used to fine-tune the classification model using new labeled data based on the original classification model, avoiding retraining the entire classification model, and improving the adaptability of the classification model.

[0077] The document classification method provided by the embodiment performs cleaning and standardization processing on the collected documents, extracts structural information and metadata, and stores the preprocessed document data in a certain format, thereby providing high-quality data support for subsequent feature extraction. The BERT model combined with the bidirectional attention mechanism is used to extract semantic features, the structural features are extracted by analyzing the document structural information, the domain features are extracted by combining the domain dictionary and the topic model, the deep semantic association of the documents can be deeply mined, the document organization architecture and importance can be reflected, and the professional characteristics of the document domain can be embodied, thereby effectively improving the expression ability and distinguishability of the features and laying a foundation for accurate classification of the classification model. The weighted fusion method based on the attention mechanism is used to linearly transform the semantic features, the structural features and the domain features, calculate the importance weights of the features in each dimension and normalize the features, and then obtain the comprehensive document feature vector, so that the features with greater contribution to classification can obtain higher weights, and the effectiveness of feature fusion is improved. The feature vector obtained after the pre-processing, feature extraction and fusion of the document to be classified is input into the trained classification model, the model outputs the probability distribution of the category to which the document belongs, the category with the maximum probability is taken as the final classification result, the knowledge base documents can be quickly and accurately classified, and the classification efficiency is improved.

[0078] On the basis of the above embodiment, Figure 2 The flowchart of another document classification method provided by the embodiment of the application is specifically as follows: Figure 2 As shown in the figure, the steps include: 1) knowledge base document data collection and preprocessing, constructing a training data set; 2) extracting a feature vector from a document sample in the training data set, obtaining a multi-dimensional feature vector; 3) performing feature fusion on the multi-dimensional feature vector, obtaining a fused feature vector; 4) taking the fused feature vector of the document sample and the category label of the document sample as the input of a classification model, training the classification model; 5) classifying a new document through the trained classification model, determining a classification result; 6) evaluating the classification result according to evaluation indexes, and optimizing the classification model according to the evaluation result; 7) determining the final classification result of the new document according to the optimized classification model, and completing the classification of the knowledge base document.

[0079] On the basis of the above embodiment, Figure 3A structural schematic diagram of a document classification system provided by an embodiment of the present application is provided, and the document classification system comprises a document data acquisition and preprocessing module, a multi-dimensional feature extraction module, a multi-dimensional feature fusion module, a classification model training module, a document classification module, and a classification result evaluation and optimization module. The document data acquisition and preprocessing module is configured to call an API interface provided by a knowledge base or a crawler technology, to acquire document data from various knowledge bases, and to perform preprocessing operations on the document data. The multi-dimensional feature extraction module is configured to extract semantic features, structural features, and domain features from the preprocessed document data. The multi-dimensional feature fusion module is configured to fuse the extracted semantic features, structural features, and domain features by using a weighted fusion strategy, to form a comprehensive document feature vector, and to dynamically adjust the weights of the features in each dimension by using an attention mechanism during the fusion process, so that the features in the dimension that contribute more to classification obtain higher weights. The classification model training module is configured to train a deep learning classification model by using a large number of document sample data with labeled categories, to prevent model overfitting by using cross-validation, early stopping, and other techniques during the training process, and to optimize the model parameters by using a dynamic learning rate adjustment strategy, to improve the classification performance of the model. The document classification module is configured to input a preprocessed document to be classified into the trained classification model, to output a probability distribution of the category to which the document belongs by the model, and to determine the final category of the document according to the probability distribution. The classification result evaluation and optimization module is configured to evaluate the performance of the classification results by using accuracy, recall rate, F1 value, and other indicators, to analyze the reasons for the classification errors of the document categories according to the evaluation results, and to optimize the parameters involved in the multi-dimensional feature extraction module and the classification model training module, to continuously improve the classification accuracy.

[0080] For example, see Figure 4 , Figure 4 Another structural schematic diagram of a document classification system provided by an embodiment of the present application is provided, wherein the document data acquisition and preprocessing comprises data cleaning, text standardization, structural information extraction, and metadata extraction. The multi-dimensional feature extraction module comprises semantic feature extraction, structural feature extraction, and domain feature extraction. The semantic features are extracted by using a BERT model combined with a bidirectional attention mechanism, which can capture the deep semantic association of the document. The structural features are extracted by analyzing the structural information such as the title, abstract, and chapter hierarchy of the document, which can reflect the organization architecture and importance of the document. The domain features are extracted by using a domain dictionary and a topic model, which can reflect the professional characteristics of the domain to which the document belongs.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software and a general hardware platform as required, of course, it can also be implemented by hardware, but in many cases the former is a better embodiment.

[0082] The embodiment of the application further provides a document classification device, comprising: an information extraction unit configured to perform information extraction on the document to be classified from multiple dimensions to obtain multiple document information; a feature extraction unit configured to perform feature extraction on the multiple document information to obtain multiple dimension feature vectors, wherein each document information corresponds to a dimension feature vector; a calculation unit configured to calculate the importance weight of the multiple dimension feature vectors according to an activation function and a set attention weight parameter, and perform weighted fusion on the multiple dimension feature vectors and the corresponding importance weight to obtain a document feature vector; a classification unit configured to perform document classification on the document to be classified based on the document feature vector through a pre-trained classification model to obtain a document category of the document to be classified.

[0083] Optionally, the information extraction unit is configured to: perform standardization processing on the document to be classified to obtain text content information, wherein the standardization processing comprises word segmentation processing; analyze the format of the document to be classified, determine at least one logical structure, and record the position and hierarchical relationship of the at least one logical structure in the document to be classified to obtain document structure information; extract attribute information of the document to be classified to obtain document attribute information; wherein the multiple document information comprises at least one content information in the text content information, the document structure information and the document attribute information.

[0084] Optionally, the feature extraction unit is configured to: perform semantic encoding on the text content information to obtain deep semantics to obtain a semantic feature vector; extract a structure feature vector based on the document structure information, wherein the structure feature vector is used to represent the organization mode and importance of the document; extract a domain feature vector from the document attribute information through a pre-constructed domain dictionary and a pre-trained topic model; wherein the multiple dimension feature vectors comprise at least one feature vector in the semantic feature vector, the structure feature vector and the domain feature vector.

[0085] Optionally, the feature extraction unit is configured to: segment and encode the text content information according to the input requirements of a pre-trained language model to generate a word embedding vector of each word, wherein the text content information comprises multiple words; extract a context-related vector representation of each word based on the word embedding vector of each word through a language model to obtain a word vector of each word; The first attention weight between different words is calculated based on the word vector of each word, wherein the attention weight is used to represent the weight of different words in semantic association; The semantic feature vector is obtained by weighted summation of the word vectors according to the first attention weight.

[0086] The plurality of words include a first word and a second word.

[0087] Optionally, the feature extraction unit is configured to: The product of the word vector of the first word, the word vector of the second word and the first set weight matrix is calculated to obtain the association score between the first word and the second word; The sum value of the association scores between the first word and other words in the plurality of words except the first word is calculated, wherein the other words include the second word; The ratio of the association score and the sum value is calculated to obtain the first attention weight between the first word and the second word.

[0088] Optionally, the feature extraction unit is configured to: For each word, the word vector of each word is weighted and summed according to the attention weight between each word and other words in the plurality of words except each word to obtain a sentence vector at a sentence level; The second attention weight between different sentence vectors is calculated; The semantic feature vector at a document level is obtained by weighted summation of the sentence vectors according to the second attention weight.

[0089] The document structure information includes at least one of document title information, document abstract information, chapter title information and chapter paragraph information.

[0090] Optionally, the feature extraction unit is configured to: The document title information is encoded into a title feature vector by a pre-trained language model; The document abstract information is encoded into an abstract feature vector by a language model; The chapter title information and the chapter paragraph information are respectively encoded by a language model, and the structural features corresponding to the chapter title information and the chapter paragraph information are fused by an attention mechanism to obtain a chapter vector; The chapter feature vector is obtained by weighted summation of all chapter vectors according to a set chapter weight, wherein the chapter weight is set according to the level of the chapter in the document; The structural feature vector is obtained based on the title feature vector, the abstract feature vector and the chapter feature vector.

[0091] Optionally, the feature extraction unit is configured to: obtaining at least one pre-constructed field dictionary, wherein the field dictionary comprises at least one core term of a corresponding field; calculating a frequency of the at least one core term included in each field dictionary in the document attribute information to obtain a field term frequency vector; calculating, by a pre-trained topic model, a probability distribution of the document to be classified on at least one set topic corresponding to the topic model based on the document attribute information to obtain a topic feature vector; obtaining a field feature vector according to the field term frequency vector and the topic feature vector.

[0092] Optionally, the computing unit is configured to: linearly transforming each dimension feature vector according to the second set weight matrix and the set bias term to unify the dimension feature vectors; calculating an importance weight of the linearly transformed dimension feature vector according to the activation function and the set attention weight parameter; normalizing the importance weight of the linearly transformed dimension feature vector to obtain a normalized weight of the linearly transformed dimension feature vector; calculating the fused document feature vector according to the normalized weight of the linearly transformed dimension feature vector and the linearly transformed dimension feature vector.

[0093] Optionally, the document classification device further comprises a training unit, and the training unit is configured to: constructing a training data set, wherein the training data set comprises a document category label of a document sample and a fused feature vector of the document sample, and the fused feature vector is obtained by weighting and fusing multi-dimensional features of the document sample; initializing parameters of a classification model, taking the fused feature vector as an input of the classification model, and outputting a document predicted category of the document sample; calculating a loss value based on the document predicted category and the document category label by using a weighted cross-entropy loss function, and updating model parameters of the classification model according to the loss value.

[0094] The training data set comprises a training set and a validation set, the training set is used for training the classification model, and the validation set is used for verifying the classification model trained based on the training set.

[0095] Optionally, the training unit is configured to: calculating loss values of a plurality of training batches; if a loss value reduction amplitude of a continuous preset number of training batches is less than a preset threshold, stopping training of the classification model.

[0096] Optionally, the training unit is configured to: The classification results of the classification model are evaluated according to the set evaluation index, and an evaluation result is obtained. Reasons for classification errors of the classification model for the target document are analyzed according to the evaluation result, and an analysis result is obtained. Based on the analysis result, the number of training samples of the target document is increased, the feature weight of the target category document is adjusted, and / or the classification model is fine-tuned by using an incremental learning strategy.

[0097] It can be understood that the description of the features in the embodiments of the document classification device can refer to the related description of the embodiments of the document classification method, which will not be repeated here.

[0098] Embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above document classification method embodiments.

[0099] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above document classification method embodiments when running.

[0100] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0101] Embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above document classification method embodiments.

[0102] Embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above document classification method embodiments.

[0103] Those skilled in the art will further realize that the mere conception of the examples described herein is sufficient to enable practitioners to practice the examples as changed or modified for variations of the application, and are further real ized that the following examples are illustrative only and not limiting of the scope of the present application. Accordingly, professional artisans understand that numerous changes can be made and still obtain a rewarding and useful result as changes are suitably made within the scope of the concepts disclosed and illustrated herein, and that only the illustrated and described examples are not to be construed as limiting of the scope of the present application.

[0104] The above has carried on the detailed introduction to the document classification provided by the application. The principle and implementation of the application are described by applying specific examples in this paper. The above example description is only applicable to help understand the method and core idea of the application. It should be pointed out that for ordinary skilled in the art, some improvements and modifications can be made to the application without departing from the principle of the application. These improvements and modifications also fall within the protection scope of the claims of the application.

Claims

1. A method of classifying a document, characterized by, The method comprises the following steps: information extraction is performed on the to-be-classified document from multiple dimensions to obtain multiple document information; feature extraction is performed on the multiple document information to obtain multiple dimension feature vectors, wherein each document information corresponds to a dimension feature vector; importance weights of the multiple dimension feature vectors are calculated according to an activation function and a set attention weight parameter, and the multiple dimension feature vectors and the corresponding importance weights are weighted and fused to obtain a document feature vector; a document classification model that is pre-trained is used to perform document classification based on the document feature vector to obtain a document category of the to-be-classified document.

2. The method of claim 1, wherein, The information extraction is performed on the to-be-classified document from multiple dimensions to obtain multiple document information, which comprises the following steps: standardization processing is performed on the to-be-classified document to obtain text content information, wherein the standardization processing comprises word segmentation processing; a format of the to-be-classified document is parsed to determine at least one logical structure, and a position and a hierarchical relationship of the at least one logical structure in the to-be-classified document are recorded to obtain document structure information; attribute information of the to-be-classified document is extracted to obtain document attribute information; The multiple document information comprises at least one content information in the text content information, the document structure information and the document attribute information.

3. The method of claim 2, wherein, The feature extraction is performed on the multiple document information to obtain multiple dimension feature vectors, which comprises the following steps: semantic encoding is performed on the text content information to obtain deep semantics to obtain a semantic feature vector; a structure feature vector is extracted based on the document structure information, wherein the structure feature vector is used to represent an organization mode and an importance of the document; a domain feature vector is extracted from the document attribute information through a pre-constructed domain dictionary and a pre-trained topic model; The multiple dimension feature vectors comprise at least one feature vector in the semantic feature vector, the structure feature vector and the domain feature vector.

4. The method of claim 3, wherein, The semantic encoding is performed on the text content information to obtain deep semantics to obtain a semantic feature vector, which comprises the following steps: segmentation and encoding are performed on the text content information according to input requirements of a pre-trained language model to generate a word embedding vector of each word, wherein the text content information comprises multiple words; a context-related vector representation of each word is extracted based on the word embedding vector of each word through the language model to obtain a word vector of each word; a first attention weight between different words is calculated based on the word vector of each word, wherein the attention weight is used to represent a weight of different words in semantic correlation; a weighted sum of the word vectors is performed according to the first attention weight to obtain a semantic feature vector.

5. The method of claim 4, wherein, The multiple words comprise a first word and a second word, and the first attention weight between different words is calculated based on the word vector of each word, which comprises the following steps: a product of the word vector of the first word, the word vector of the second word and a first set weight matrix is calculated to obtain a correlation score between the first word and the second word; calculating a sum value of the association scores between the first word and other words in the plurality of words except the first word, wherein the other words include the second word; calculating a ratio of the association score and the sum value to obtain a first attention weight between the first word and the second word.

6. The method of claim 4, wherein, The weighting and summing of the word vectors according to the first attention weight to obtain the semantic feature vector comprises: For each word, weighting and summing the word vector of each word according to the attention weight between each word and other words in the plurality of words except each word to obtain a sentence-level sentence vector; calculating a second attention weight between different sentence vectors; weighting and summing the sentence vectors according to the second attention weight to obtain a document-level semantic feature vector.

7. The method of claim 3, wherein, The document structure information includes at least one of the document title information, the document abstract information, the chapter title information and the chapter paragraph information, and the extraction of the structural feature vector based on the document structure information comprises: encoding the document title information into a title feature vector through a pre-trained language model; encoding the document abstract information into an abstract feature vector through the language model; encoding the chapter title information and the chapter paragraph information through the language model respectively, and fusing the corresponding structural features of the chapter title information and the chapter paragraph information through an attention mechanism to obtain a chapter vector; weighting and summing all chapter vectors according to a set chapter weight to obtain a chapter feature vector, wherein the chapter weight is set according to the level of the chapter in the document; obtaining a structural feature vector based on the title feature vector, the abstract feature vector and the chapter feature vector.

8. The method of claim 3, wherein, The extraction of the domain feature vector from the document attribute information based on the pre-constructed domain dictionary and the pre-trained topic model comprises: obtaining at least one pre-constructed domain dictionary, wherein the domain dictionary includes at least one core term corresponding to a domain; calculating the frequency of occurrence of at least one core term included in each domain dictionary in the document attribute information to obtain a domain term frequency vector; calculating the probability distribution of the to-be-classified document on at least one set topic corresponding to the topic model based on the document attribute information through a pre-trained topic model to obtain a topic feature vector; obtaining a domain feature vector according to the domain term frequency vector and the topic feature vector.

9. The method of claim 1, wherein, The calculation of the importance weight of the plurality of dimension feature vectors according to the activation function and the set attention weight parameter, and the weighting and fusion of the plurality of dimension feature vectors and the corresponding importance weight to obtain the document feature vector comprise: linearly transforming each dimension feature vector according to a second set weight matrix and a set bias term to unify the dimension feature vectors; calculating the importance weight of the linearly transformed dimension feature vectors according to the activation function and the set attention weight parameter; normalize importance weights of the linearly transformed feature vectors of each dimension to obtain normalized weights of the linearly transformed feature vectors of each dimension; calculate a fused document feature vector according to the normalized weights of the linearly transformed feature vectors of each dimension and the linearly transformed feature vectors of each dimension.

10. The method of claim 1, wherein, The training process of the classification model is as follows: construct a training data set, wherein the training data set includes document category labels of document samples and fused feature vectors of the document samples, wherein the fused feature vectors are obtained by weighted fusion of multi-dimensional features of the document samples; initialize parameters of the classification model, and input the fused feature vectors as inputs of the classification model to output document predicted categories of the document samples; use a weighted cross-entropy loss function to calculate a loss value based on the document predicted categories and the document category labels, and update model parameters of the classification model according to the loss value.

11. The method of claim 10, wherein, The training data set includes a training set and a validation set, the training set is used to train the classification model, and the validation set is used to verify the classification model trained based on the training set. In the process of verifying the classification model based on the validation set, the method further comprises: calculate loss values of multiple training batches; if a loss value reduction amplitude of a continuous preset number of training batches is less than a preset threshold, stop training the classification model.

12. The method of claim 10, wherein, After updating the model parameters of the classification model according to the loss value, the method further comprises: evaluate classification results of the classification model according to a set evaluation index to obtain an evaluation result; analyze reasons for classification errors of the classification model for target documents according to the evaluation result to obtain an analysis result; based on the analysis result, increase a training sample number of the target documents, adjust feature weights of the target documents, and / or use an incremental learning strategy to fine-tune the classification model.

13. An electronic device, comprising: comprise: a memory for storing a computer program; a processor for implementing steps of the document classification method of any one of claims 1 to 12 when executing the computer program.

Images