Document classification method and system, computer equipment and storage medium

Abstract

The invention provides a document classification method, which belongs to the field of data classification, and comprises the following steps: collecting original documents of different languages from a plurality of channels, and converting texts of different languages into a unified language representation form; extracting keywords in the document after language conversion, calculating word frequency and inverse document frequency of each keyword in the corresponding document, weighting the word frequency by adopting the inverse document frequency to obtain a plurality of TF-IDF weights, and converting the plurality of TF-IDF weights into a plurality of word embedding features; meanwhile, text data similar to the embedding features of the multiple words in the original document are labeled; inputting the plurality of word embedding features into a natural language processing model to train the model, and training the trained model again through the labeled document data to obtain a semantic encoder; and inputting the to-be-classified documents of different languages into a semantic encoder, and outputting a classification result. According to the method, the documents of different languages can be accurately classified by adopting two hierarchical training models, so that the document classification efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data classification, and specifically relates to a document classification method, system, computer equipment and storage medium. Background Art

[0002] With the rapid development of information technology, the number of documents has increased dramatically. How to efficiently manage and classify these documents has become an urgent problem to be solved. Traditional document classification methods rely on manual classification, which is not only inefficient but also prone to errors.

[0003] Therefore, currently, machine learning, neural network models, etc. are usually used to extract document features, classify documents based on the extracted feature results, and classify documents based on the classification results. However, the training data used in the training of existing models is usually single language data that needs to be classified. Due to the scope of training data limitations, the trained model is usually not universal, which reduces the accuracy of model classification. The model needs to be retrained for documents of different language types, which reduces the efficiency of the entire document classification. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a document classification method, system, computer device and storage medium.

[0005] In order to achieve the above object, the present invention provides the following technical solutions: A document classification method comprises the following steps: Collect original documents in different languages ​​from multiple channels, process them through a cross-language pre-trained model, and convert texts in different languages ​​into a unified language representation; Extract keywords from the document after language conversion, calculate the word frequency TF and inverse document frequency IDF of each keyword in the corresponding document, use the inverse document frequency IDF to weight the word frequency TF to obtain multiple TF-IDF weights, and convert the multiple TF-IDF weights into multiple word embedding features; at the same time, annotate the text data in the original document that are similar to the multiple word embedding features; Input multiple word embedding features into the natural language processing model to train the model, and then retrain the trained natural language processing model with the annotated document data to obtain a semantic encoder; The documents in different languages ​​to be classified are input into the semantic encoder, and the corresponding classification labels are output.

[0006] Preferably, the collecting of original documents in different languages ​​from multiple channels specifically includes: collecting original documents in different languages ​​from an internal database of the enterprise, the Internet and a partner database.

[0007] Preferably, before extracting keywords from the document after language conversion, the original document is preprocessed to extract keywords from the preprocessed document, and the preprocessing includes: Remove noise data from the document after language conversion, and perform standardization on the document after noise removal; Perform word segmentation on the standardized documents to divide continuous documents into words or phrases; Remove stop words from words or phrases.

[0008] Preferably, the noise data includes HTML tags, JavaScript codes, comments, special characters, URL links and advertisements; the standardization of the document specifically includes: unifying the encoding format, punctuation marks and capitalization of the document.

[0009] Preferably, the natural language processing model is any one of a support vector machine model SVM, a convolutional neural network CNN, a recurrent neural network RNN, a long short-term memory network LSTM, or a deep learning model Transformer.

[0010] Preferably, an optimization algorithm such as gradient descent is used to minimize the loss function, and the model parameters are updated through a back-propagation algorithm.

[0011] Preferably, a pre-trained word embedding model is used to convert multiple TF-IDF weights into multiple word embedding features.

[0012] The present invention also provides a document classification system, comprising: The data acquisition module is used to collect original documents in different languages ​​from multiple channels, process the original documents through the cross-language pre-training model, and convert texts in different languages ​​into a unified language representation; The data processing module is used to extract keywords from the document after language conversion, calculate the word frequency TF and inverse document frequency IDF of each keyword in the corresponding document, use the inverse document frequency IDF to weight the word frequency TF to obtain multiple TF-IDF weights, and convert the multiple TF-IDF weights into multiple word embedding features; at the same time, the text data in the original document that is similar to the multiple word embedding features is annotated; A model training module is used to input multiple word embedding features into a natural language processing model to train the model, and then retrain the trained natural language processing model with the annotated document data to obtain a semantic encoder; The document classification module is used to input documents in different languages ​​to be classified into the semantic encoder and output corresponding classification labels.

[0013] The present invention also provides a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement any one of the steps in the document classification method.

[0014] The present invention also provides a computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is loaded by a processor, it can execute any one of the steps in the document classification method.

[0015] The document classification method provided by the present invention has the following beneficial effects: The present invention first collects documents in different languages ​​from multiple channels, increases the scope of model training data, processes the original documents through a cross-language pre-training model, converts texts in different languages ​​into a unified language representation, and can assist in accurately capturing the commonalities and differences between different languages, thereby improving the accuracy of cross-language classification; at the same time, two levels of pre-training are adopted, firstly, the language model is trained through unlabeled document data so that it can learn the common sense of the target language. Then, the semantic encoder is further trained on the basis of the trained language model using labeled data. This deep pre-training method enables the model to better understand the semantic information of the document. The trained model can learn the ability to semantically encode documents, that is, the more similar the documents are, the closer the semantic vectors obtained after encoding are in space, and vice versa, the farther they are in space. This ability makes document classification more accurate. The model trained with two levels can accurately classify documents in different languages, improving the efficiency of the entire document classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a flowchart of a document classification method according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a document classification system according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0019] To enable those skilled in the art to better understand the technical solution of the present invention and be able to implement it, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and cannot be used to limit the protection scope of the present invention.

[0020] Embodiment 1 The present invention provides a document classification system, specifically as Figure 1 shown, including the following steps: S1. Document collection.

[0021] Collect a large number of original documents in different languages that need to be classified from multiple channels (such as enterprise internal databases, the Internet, partners, etc.).

[0022] S2. Data preprocessing.

[0023] 1. Data cleaning: Remove the noise data in the original documents, such as HTML tags, JavaScript code, comments, special characters, URL links, and advertisements, etc. At the same time, perform standardization processing on the denoised documents, such as unifying the encoding format, punctuation marks, capitalization, etc.

[0024] 2. Word segmentation and stop word processing.

[0025] Word segmentation: Use professional word segmentation tools (such as jieba segmentation, HanLP, etc.) to perform word segmentation on the cleaned documents, and cut the continuous documents into words or phrases. The word segmentation result directly affects the subsequent feature extraction and model training effect.

[0026] Stop word processing: Remove the stop words that are useless for classification in the words or phrases after word segmentation, such as common words like "de", "shi", "zai", etc. These words appear frequently in the documents but do not make a substantial contribution to the theme and category of the documents.

[0027] 3. Feature extraction TF-IDF feature: extract keywords from the preprocessed documents, calculate the term frequency (TF) and inverse document frequency (IDF) of each keyword in the corresponding document, and use the inverse document frequency (IDF) to weight the term frequency TF to obtain multiple TF-IDF weights as the feature representation of the document. The TF-IDF value can reflect the importance and uniqueness of a word in a document. Term frequency TF refers to the frequency of a word in a document. Generally speaking, the more times a word appears in a document, the more important it is in the document. Inverse document frequency (IDF) is used to measure the prevalence of a word in the entire corpus. If a word appears in many documents, its inverse document frequency will be very low, otherwise it will be very high. The inverse document frequency is calculated by dividing the total number of documents by the number of documents containing the word and then taking the logarithm. The TF-IDF calculation formula is to multiply the term frequency and the inverse document frequency, that is, TF x IDF. The purpose of this is to combine the importance of a word in a document with its prevalence in the entire corpus to obtain a more accurate measurement value.

[0028] Word embedding features: Use pre-trained word embedding models (such as Word2Vec, GloVe, BERT, etc.) to convert multiple TF-IDF weights into word embedding features. Word embedding features can capture the semantic relationship between words and help improve the performance of classification models. At the same time, the text data in the original document that is similar to multiple word embedding features is annotated.

[0029] 4. Data Partitioning Divide the word embedding features into training set, validation set and test set. Usually, the training set is used for model training, the validation set is used for model parameter adjustment and to prevent overfitting, and the test set is used to evaluate model performance. The data division ratio can be adjusted according to the specific task, but it is generally recommended that the training set accounts for the largest proportion, and the validation set and test set account for a relatively small proportion.

[0030] S3. Model selection and training 1. Model selection Select an appropriate machine learning or deep learning model as the natural language processing model based on the specific task characteristics of document classification (such as document length, number of categories, category balance, etc.). Common models include Naive Bayes, Support Vector Machine (SVM), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), Transformer, etc.

[0031] 2. Model Training Multiple word embedding features are input into the natural language processing model to train the model, and then the trained natural language processing model is retrained with the annotated document data to obtain a semantic encoder. During the training process, optimization algorithms such as gradient descent are used to minimize the loss function, and the model parameters are updated through the back propagation algorithm. At the same time, strategies such as early stopping, learning rate decay, and dropout are used to prevent overfitting.

[0032] Monitor performance indicators on the validation set (such as accuracy, recall, F1 score, etc.) to adjust model parameters and optimize model structure in a timely manner.

[0033] 3. Performance Evaluation Use the test set to evaluate the classification performance of the model. Evaluation indicators include accuracy, precision, recall, F1 score, confusion matrix, etc. These indicators can fully reflect the classification effect of the model in different categories.

[0034] S4. Document classification and evaluation 1. Document classification The documents in different languages ​​to be classified are input into the semantic encoder, and the corresponding classification labels are output.

[0035] Sort or filter the classification results to meet the needs of different scenarios. For example, in the news classification scenario, you can sort the news in chronological order; in the knowledge management scenario, you can filter the documents by keywords.

[0036] 2. Model Evaluation Further optimize or adjust the model based on the evaluation results. For example, adjust model parameters, change feature extraction methods, try different model structures, etc.

[0037] Based on the above document classification method provided by the present invention, the following operations can also be performed: 1. System deployment Deploy the trained model to the server and build an automatic document classification system. The system should support high concurrency processing and high availability to ensure that it can quickly respond to requests from a large number of users.

[0038] Design a user-friendly front-end interface to facilitate users to upload documents and view classification results.

[0039] 2. Application expansion Apply document automatic classification technology to practical scenarios, such as enterprise document management, university document archiving, library book classification, etc. By automatically processing a large number of documents, the efficiency and quality of document management can be improved.

[0040] Continuously iterate and optimize system performance based on user feedback and needs. For example, new feature extraction methods can be introduced, model structures can be improved, new classification categories can be added, etc.

[0041] In summary, the detailed and specific technical solutions of document automatic classification technology cover multiple links such as data preprocessing, model selection and training, document classification and evaluation, and system deployment and application. By continuously optimizing and improving each step and method in these links, an efficient, accurate and stable document automatic classification system can be built to provide convenient document management services for all walks of life.

[0042] The document classification method provided by the present invention can be applied in multiple fields, and specific examples are provided as follows: Enterprise document management: A large manufacturing company uses automatic document classification technology to manage its internal documents. Through this technology, the company successfully classified tens of thousands of documents into different folders, including product manuals, technical reports, sales contracts, etc. This not only improves the efficiency of document management, but also makes it easier for employees to quickly find relevant documents when needed.

[0043] University document archiving: A university uses document automatic classification technology to archive its official documents. This technology can automatically identify the type and subject of official documents and classify them into corresponding folders. This greatly reduces the workload of manual classification and improves the accuracy and efficiency of archiving. At the same time, this technology also supports rapid retrieval and query of archived files, providing strong support for university work.

[0044] Library book classification: A university library uses document automatic classification technology to classify and manage its collection of books. This technology can automatically identify the subject and category of books and classify them into corresponding bookshelves or tags. This improves the retrieval efficiency and borrowing convenience of books, and provides readers with a better service experience.

[0045] The present invention has achieved a variety of significant effects in practical applications, which not only improve the efficiency and accuracy of document management, but also bring many conveniences to enterprises, universities and other institutions. The following are some specific effects and examples:

[0046] 1. Improve document management efficiency Automated processing: Automatic document classification technology can automatically identify and classify a large number of documents without manual intervention. This greatly reduces the time and labor costs required for manual classification and improves the efficiency of document management. For example, in an enterprise, hundreds or thousands of documents may need to be processed every day. Through automatic classification technology, these documents can be accurately classified into corresponding folders or tags in a short time.

[0047] Quick search: Automatically classified documents can be quickly searched based on classification tags. Users only need to enter keywords or select classification tags to quickly find the required documents. This greatly improves the efficiency of document retrieval and saves users' time.

[0048] 2. Improve classification accuracy Intelligent identification: Automatic document classification technology uses machine learning or deep learning algorithms to intelligently identify the content and features of documents, thereby accurately classifying them. This intelligent identification capability is more accurate and reliable than traditional manual classification. For example, in the legal field, automatic document classification technology can accurately identify the type and subject of legal documents, helping lawyers quickly find relevant cases and regulations.

[0049] Reduce errors: Errors and omissions are inevitable in the manual classification process, while automatic classification technology uses algorithms and models to classify, reducing errors caused by human factors. This improves the accuracy and reliability of classification and ensures the correct classification of documents.

[0050] 3. Optimize user experience Personalized recommendations: In social media or e-commerce platforms, automatic document classification technology can also combine user behavior and preferences to make personalized recommendations. By analyzing information such as user browsing history and purchase records, the system can automatically recommend documents or products that users may be interested in. This improves user experience and satisfaction.

[0051] Convenient operation: Automatic classification technology is usually combined with a user-friendly interface to provide a convenient operation method. Users only need to upload documents or enter keywords to obtain classification results. This convenience makes users more willing to use automatic classification systems to process documents.

[0052] A technical difference from existing automatic document classification technologies is the cross-language document classification. This technology breaks through the limitation of traditional document classification technologies that mainly focus on a single language and realizes the automatic classification and processing of multi-language documents.

[0053] The characteristics of cross-language document classification technology are as follows: 1. Multi-language understanding capability: This technology can understand and analyze document content in different languages, including but not limited to Chinese, English, French, Spanish, etc. This requires the system to have strong multi-language processing capabilities and cross-language semantic understanding capabilities.

[0054] 2. Unified representation and classification: The system can convert document content in different languages ​​into a unified representation (such as vector space model, word embedding, etc.) so that it can be compared and classified under a unified framework. This conversion process needs to fully consider the semantic differences and grammatical structures between different languages.

[0055] 3. Cross-language model training: In order to achieve cross-language document classification, the system needs to use cross-language pre-trained models (such as mBERT, XLM-RoBERTa, etc.). These models are trained on a large number of multilingual corpora and can capture the commonalities and differences between different languages, thereby improving the accuracy of cross-language classification.

[0056] 4. Domain adaptability: Cross-language document classification technology also needs to consider the specificity and professionalism of documents in different languages. The system can be optimized and adjusted for documents in specific fields through domain adaptation training to improve the accuracy and practicality of classification.

[0057] Application scenario examples: Document management for multinational companies: For multinational companies, their internal documents may involve multiple languages. Through cross-language document classification technology, companies can automatically classify documents in different languages ​​into corresponding folders or tags to improve the efficiency and accuracy of document management.

[0058] Multilingual document retrieval: There are a large number of multilingual documents in libraries or academic institutions. Cross-language document classification technology can help users quickly find the Chinese, English or other language versions of the required documents, improving the efficiency of document retrieval and user experience.

[0059] Cross-border e-commerce product description classification: On cross-border e-commerce platforms, product descriptions may involve multiple languages. Through cross-language document classification technology, the platform can automatically classify product descriptions into corresponding product categories, making it easier for consumers to find and purchase.

[0060] Cross-language document classification technology is an important technical point that is different from existing document automatic classification technology. It realizes the automatic classification and processing of multi-language documents through its multi-language understanding ability, unified representation and classification, cross-language model training and domain adaptability, bringing convenience and benefits to cross-language document management, document retrieval and cross-border e-commerce.

[0061] Based on the same inventive concept, the present invention also provides a document classification method system, such as Figure 2As shown, including: The data acquisition module 101 is used to collect original documents in different languages ​​from multiple channels, process the original documents through a cross-language pre-training model, and convert texts in different languages ​​into a unified language representation.

[0062] The data processing module 102 is used to extract keywords from the document after language conversion, calculate the term frequency TF and the inverse document frequency IDF of each keyword in the corresponding document, use the inverse document frequency IDF to weight the term frequency TF to obtain multiple TF-IDF weights, and convert the multiple TF-IDF weights into multiple word embedding features; at the same time, mark the text data in the original document that is similar to the multiple word embedding features.

[0063] The model training module 103 is used to input multiple word embedding features into the natural language processing model to train the model, and then re-train the trained natural language processing model with the annotated document data to obtain a semantic encoder.

[0064] The document classification module 104 is used to input documents in different languages ​​to be classified into a semantic encoder and output corresponding classification labels.

[0065] Each module in the document classification method system can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0066] The present invention also provides a computer device, including a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the document classification method embodiment. The specific implementation method can be found in the method embodiment, which will not be described here.

[0067] Furthermore, the present invention also provides a non-temporary computer-readable storage medium containing instructions, and a computer program is stored on the storage medium. For example, a memory containing instructions, the above instructions can be executed by a processor of a computer device to complete the above method. For example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in the document classification method embodiment can be implemented. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0068] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] It should be pointed out that the specific implementation methods described above can enable those skilled in the art to understand the invention more comprehensively, but do not limit the invention in any way. Therefore, although the invention has been described in detail in this specification and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents; and all technical solutions and improvements that do not deviate from the spirit and scope of the invention are included in the protection scope of the patent for the invention. Any figure mark in the claims should not be regarded as limiting the claims involved. Any simple change or equivalent replacement of the technical solution that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belongs to the protection scope of the present invention.

Claims

1. A document classification method, characterized in that: The following steps are involved: Collect original documents in different languages ​​from multiple channels, process them through a cross-language pre-trained model, and convert texts in different languages ​​into a unified language representation; Extract keywords from the document after language conversion, calculate the word frequency TF and inverse document frequency IDF of each keyword in the corresponding document, use the inverse document frequency IDF to weight the word frequency TF to obtain multiple TF-IDF weights, and convert the multiple TF-IDF weights into multiple word embedding features; At the same time, the text data in the original document that is similar to the multiple word embedding features is annotated; Input multiple word embedding features into the natural language processing model to train the model, and then retrain the trained natural language processing model with the annotated document data to obtain a semantic encoder; The documents in different languages ​​to be classified are input into the semantic encoder, and the corresponding classification labels are output.

2. The document classification method according to claim 1, characterized in that: The collecting of original documents in different languages ​​from multiple channels specifically includes: collecting original documents in different languages ​​from the enterprise's internal database, the Internet and the partner database.

3. The document classification method according to claim 1, characterized in that: Before extracting keywords from the document after language conversion, the original document is preprocessed to extract keywords from the preprocessed document. The preprocessing includes: Remove noise data from the document after language conversion, and perform standardization on the document after noise removal; Perform word segmentation on the standardized documents to divide continuous documents into words or phrases; Remove stop words from words or phrases.

4. The document classification method according to claim 3, characterized in that: The noise data includes HTML tags, JavaScript codes, comments, special characters, URL links and advertisements; the standardization of the document specifically includes: unifying the encoding format, punctuation marks and capitalization of the document.

5. The document classification method according to claim 3, characterized in that: The natural language processing model is any one of a support vector machine model SVM, a convolutional neural network CNN, a recurrent neural network RNN ​​or a long short-term memory network LSTM or a deep learning model Transformer.

6. The document classification method according to claim 5, characterized in that: During the model training process, the gradient descent optimization algorithm is used to minimize the loss function, and the back-propagation algorithm is used to update the model parameters.

7. The document classification method according to claim 1, characterized in that: Use a pre-trained word embedding model to convert multiple TF-IDF weights into multiple word embedding features.

8. A document classification system, characterized in that: include: The data acquisition module is used to collect original documents in different languages ​​from multiple channels, process the original documents through the cross-language pre-training model, and convert texts in different languages ​​into a unified language representation; The data processing module is used to extract keywords from the document after language conversion, calculate the word frequency TF and inverse document frequency IDF of each keyword in the corresponding document, use the inverse document frequency IDF to weight the word frequency TF to obtain multiple TF-IDF weights, and convert the multiple TF-IDF weights into multiple word embedding features; At the same time, the text data in the original document that is similar to the multiple word embedding features is annotated; A model training module is used to input multiple word embedding features into a natural language processing model to train the model, and then retrain the trained natural language processing model with the annotated document data to obtain a semantic encoder; The document classification module is used to input documents in different languages ​​to be classified into the semantic encoder and output corresponding classification labels.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is loaded into a processor, the computer program can execute the steps of the method according to any one of claims 1 to 7.

Images