Chinese literature classification method based on incremental learning

Abstract

The present invention provides a Chinese document classification method based on incremental learning, which specifically includes: constructing a benchmark data set for Chinese document classification problems; constructing a benchmark model for Chinese document classification based on incremental learning; using a flock selection algorithm to select representative data from old categories and merge it with new category data to construct an incremental learning data set; inputting the incremental learning data set into the benchmark model and training it under the constraints of the incremental learning loss function; using a weight alignment method to update the weight of the new category output by the fully connected layer in the benchmark model; and finally inputting the Chinese document to be classified into the model after the weight alignment of the fully connected layer, performing text information recognition, and obtaining a document classification result. The present invention can be used to identify the categories of Chinese documents. Using the weight alignment method and the decoupled distillation loss function, the results of multi-classification problems are made more reliable and credible, and the system error is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of Chinese document classification, and in particular relates to an effective Chinese document classification method based on incremental learning. Background Art

[0002] Documents are essential vehicles for recording knowledge and information, and are crucial for academic research, knowledge dissemination, and cultural heritage. With the continued deepening of scientific research and the increasing prominence of interdisciplinary research, the volume of documents is experiencing explosive growth. Document classification refers to the practice of organizing and managing documents based on their content and format. This classification method helps systematically reveal and organize documents, making them easier for users to find and utilize.

[0003] Currently, document classification can be divided into two approaches: manual classification and automated classification. Manual classification relies on expert knowledge and experience to ensure accurate and targeted classification. However, faced with such a massive amount of data, traditional manual classification methods also have several drawbacks. Manual classification not only requires a significant investment of time and human resources, but is also costly. Due to human factors, errors and omissions may occur during the classification process, making it difficult to ensure consistent and accurate classification. Therefore, the development of efficient automated classification technology has become an urgent need.

[0004] As an important vehicle for reflecting core research topics, scientific literature often uses language that is highly specialized, complex, and technically sophisticated. This complexity makes automatically understanding and classifying these texts a challenging task. With the rapid development of machine learning and deep learning, the introduction of these technologies to achieve automated literature classification has important practical implications for improving both classification efficiency and accuracy.

[0005] Traditional methods for Chinese literature classification primarily rely on machine learning. Yang Min et al. used TF-IDF as a feature representation for documents and constructed an automatic bibliographic classification system based on support vector machines (SVMs). They demonstrated the feasibility of applying machine learning algorithms to the automatic classification of information resources in library practice. Li Xiangdong et al. used word frequency and word count as document features and used the K-nearest neighbor (KNN) algorithm to automatically classify journal articles belonging to different categories, achieving some success in the subject classification of journals with a small number of categories. While traditional machine learning has performed well, it still has some limitations: The encoding methods of machine learning algorithms struggle to accurately capture the deep meaning and key features of complex text, resulting in reduced classification performance; and machine learning algorithms perform poorly on large datasets, making them inadequate for processing the ever-increasing volume of literature data. In recent years, with the rise of deep learning technology, a growing number of new technologies and large pre-trained models have been applied to the field of automatic literature classification. Deng Sanhong et al. first used word embeddings to represent Chinese book texts and then used a long short-term memory (LSTM) model to construct multiple binary classifiers for multi-label classification. Luo Pengcheng et al. compared pre-trained models such as BERT and ERNIE (enhanced representation through knowledge integration) with traditional machine learning and typical deep learning algorithms on 21 first-level literature datasets in the humanities and social sciences. The results validated the superiority of the pre-trained models. However, both machine learning and deep learning-based Chinese literature classification methods currently rely on batch learning. This means that the model needs to be retrained every time new literature data arrives, further increasing the consumption of computing resources.

[0006] Based on the above background, the present invention considers using a novel method to solve the problem of the constantly emerging Chinese literature data. Incremental learning, as a machine learning paradigm that can continuously learn new knowledge, is innovatively introduced into the field of Chinese literature classification in the present invention. Incremental learning allows the model to continuously learn new sample data while retaining existing knowledge, thereby gradually improving the classification performance. This method has the following advantages: incremental learning can dynamically adapt to new data and new knowledge without retraining the entire model, thereby improving the flexibility and scalability of the classification system; compared to batch learning, incremental learning is more efficient in processing new data because it only needs to update part of the model's parameters instead of retraining the entire model; as new data is continuously added, the incremental learning model can continuously optimize its classification performance and improve the accuracy and stability of classification.

[0007] In summary, the incremental learning-based Chinese literature classification method is an innovative solution to address the challenges of document text complexity, explosive data growth, and the limitations of traditional classification methods. It is expected to revolutionize the field of document classification, significantly improving classification efficiency and accuracy, thereby more effectively promoting document management and utilization and providing important basic information support for basic scientific research. In the future, researchers will need to further study Chinese literature classification methods to achieve more comprehensive, accurate, and efficient Chinese literature classification. Summary of the Invention

[0008] To overcome the shortcomings of the aforementioned prior art, the present invention aims to provide a Chinese document classification method based on incremental learning. This method aims to create a benchmark dataset using strict filtering conditions, ensuring data accuracy and reducing system errors. A TextCNN network model is used to effectively learn local features of text sequences. The innovative introduction of an attention layer in the TextCNN network significantly improves the model's focus on key information. A model based on incremental learning is constructed, and two methods are used to effectively overcome "catastrophic forgetting," ensuring that the classification model outputs are highly accurate and reliable. One method, weight alignment (WA), corrects the bias in the fully connected layer's outputs for new and old categories caused by data imbalance by updating the weights of the fully connected layer's outputs for the new category. The other method uses a decoupled distillation loss function to help the model retain knowledge and memory of previous tasks and achieve a balance between new and old knowledge. Therefore, the advantages of this Chinese document classification method based on incremental learning include: it can adapt to the continuous growth of document data, efficiently update the model to capture new features, avoid catastrophic forgetting, and conserve computing resources. This facilitates real-time and accurate classification of Chinese documents, providing strong support for technological innovation and scientific research.

[0009] To achieve the above objectives, the main technical solution adopted by the present invention is: a Chinese document classification method based on incremental learning, comprising the following steps:

[0010] Step S1, construct a benchmark dataset for the Chinese document classification problem; Step S2, construct a benchmark model for Chinese document classification based on incremental learning; Step S3, use the flock selection algorithm to select representative data from the old category, merge it with the new category data, and construct an incremental learning dataset; Step S4, input the incremental learning dataset into the benchmark model, and train it under the constraints of the incremental learning loss function; Step S5, use the weight alignment method to update the weight of the new category of the output of the fully connected layer in the benchmark model; Step S6, input the Chinese document to be classified into the model after the weight alignment of the fully connected layer, perform sequence information recognition, and obtain the document classification result.

[0011] The beneficial effects of the present invention are:

[0012] This invention uses existing datasets to crawl data from the Baidu Academic system as a reliable data source, employing strict filtering conditions to ensure data accuracy. It also uses a pre-trained Chinese model to obtain embedded representations of Chinese document text sequences, effectively capturing the semantic information within the text. Furthermore, it applies incremental learning methods to the field of Chinese document classification, employing two approaches to address "catastrophic forgetting" and combining them with a TextCNN network model to identify Chinese document sequence information, resulting in more accurate and reliable classification results. Therefore, this invention offers the advantages of a novel approach and precise results.

[0013] In the present invention, data is crawled from the Baidu Academic system on the basis of an existing data set, and the data is subjected to strict pre-processing operations such as screening to construct a benchmark data set for the Chinese literature classification problem. The Chinese literature data set used in the present invention is a valuable resource contributed by Professor Li Ronglu of the Natural Language Processing Group of the International Database Center of the School of Computer Science and Technology of Fudan University. Baidu Academic is a powerful and resource-rich academic resource search platform that provides comprehensive, convenient and efficient academic services for scientific researchers. The present invention obtains data from the Baidu Academic system, and these data lay the foundation for the study of literature classification. The literature data obtained using crawler technology is merged with the literature data in the Fudan University Chinese Text Classification Dataset by category, and then the number of statistics and strict screening are performed. In short, through these pre-processing operations, a high-quality benchmark data set for the Chinese literature classification problem is constructed.

[0014] This paper designs an efficient baseline model for Chinese literature classification based on incremental learning. The model, consisting of an encoding layer, a convolutional layer, an activation layer, a maximum pooling layer, an attention layer, and a fully connected layer, can comprehensively and systematically identify the sequence information of Chinese literature. In particular, the present invention innovatively introduces an attention layer into the model, which assigns different weights to each feature after maximum pooling, ensuring that key features receive greater attention, thereby significantly improving the model's classification performance.

[0015] In this paper, a decoupled distillation loss combined with a cross-entropy loss is used to train the model. By adjusting the parameters of the two decoupled components of the original distillation loss, the model can better retain old knowledge while learning new knowledge. This paper also introduces a weight alignment method to overcome "catastrophic forgetting." The weights of the fully connected layer's output for new categories are updated, while the weights of the fully connected layer's output for old categories remain unchanged, resulting in a classification model with high accuracy and reliability.

[0016] The incremental learning-based Chinese literature classification method proposed in this paper uses a herd selection algorithm to select representative data from existing categories. Learning is then performed on the new category data along with representative data from existing categories. The incremental learning model learns knowledge under the constraints of an incremental learning loss function, not only learning new category information but also maintaining classification performance on test data from existing categories. This results in a more accurate, efficient, and low-memory Chinese literature classification method.

[0017] The present invention can adapt to the characteristics of the continuous growth and change of literature data, optimize the classification model by incrementally learning new data, and improve classification accuracy and efficiency. This is of great significance for quickly identifying and protecting intellectual achievements and promoting the effective use of literature information. These aspects and other aspects of the present application will be more concise and easy to understand in the description of the following embodiments. It should be understood that the above general description and the detailed description below are merely exemplary and explanatory and do not limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. Among them:

[0019] Figure 1 Schematic diagram of the steps of the Chinese document classification method based on incremental learning of the present invention;

[0020] Figure 2 This is a flowchart of the specific sub-steps of step S1 in the present invention;

[0021] Figure 3 It is a bar chart showing the number of data in each category in the Chinese literature data set downloaded and used by the present invention;

[0022] Figure 4 It is a bar chart showing the number of original data and the number of filled data for each category that needs to be filled with data when constructing a data set in the present invention;

[0023] Figure 5 Schematic diagram of the structure of the benchmark model of Chinese document classification based on incremental learning of the present invention;

[0024] Figure 6 Schematic diagram of the process of the Chinese literature classification method based on incremental learning of the present invention;

[0025] The purpose, features and advantages of this application will be further explained with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0026] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0027] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0029] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present application and are not intended to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the present application, not all of the structures.

[0030] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the accompanying drawings.

[0031] The purpose of the present invention is to address the shortcomings of the above background and propose a Chinese document classification method based on incremental learning. To achieve the above purpose, the present invention provides the following technical solutions, which are further described in detail below with reference to the accompanying drawings and examples.

[0032] The present invention provides a Chinese literature classification method based on incremental learning, such as Figure 1 As shown, the method includes the following steps:

[0033] Step S1: construct a benchmark dataset for Chinese document classification problem.

[0034] This step crawls data from the literature search system based on the existing Chinese literature dataset, and then performs strict preprocessing operations such as screening on the data to construct a high-quality benchmark dataset for Chinese literature classification problems, providing strong support for the development and application of Chinese literature classification technology.

[0035] like Figure 2As shown, step S1 may specifically include:

[0036] Step S11: Download the Chinese literature classification data set and statistically analyze the data to determine whether data needs to be filled and determine the category of data that needs to be filled.

[0037] Specifically, the Chinese literature dataset used in this invention is constructed by Professor Li Ronglu of the Natural Language Processing Group of the International Database Center of the School of Computer Science and Technology of Fudan University. This invention uses it as the original Chinese literature dataset. It can be obtained from the following link: https: / / gitcode.com / open-source-toolkit / 6a679. This dataset is designed for Chinese text classification tasks and contains a wealth of Chinese documents, involving 20 categories of Chinese literature data, and a total of 19,636 data. Figure 3 As shown in the figure, the number of data in 20 categories is shown. Among them, there are 1481 data items belonging to art, 67 data items belonging to literature, 120 data items belonging to education, 89 data items belonging to philosophy, 934 data items belonging to history, 1282 data items belonging to physical space, 65 data items belonging to energy, 55 data items belonging to electronics, 52 data items belonging to communications, 2715 data items belonging to computers, 67 data items belonging to minerals, 116 data items belonging to transportation, 2435 data items belonging to environment, 2043 data items belonging to agriculture, 3201 data items belonging to economy, 103 data items belonging to law, 104 data items belonging to medicine, 150 data items belonging to military, 2050 data items belonging to politics, and 2507 data items belonging to sports.

[0038] The above statistical results show that the 11 categories of literature, education, philosophy, energy, electronics, communications, minerals, transportation, law, pharmaceuticals, and military contain too few documents. After calculation, the category with the fewest documents is communications, accounting for 0.2648% of the total, while the category with the most documents is economics, accounting for 16.302% of the total. This indicates an extremely unbalanced data distribution. Therefore, for these 11 categories, data padding is required to balance the data across categories.

[0039] Step S12: Using the category that needs to be filled with data obtained in step S11 as a search condition, perform a search in an academic literature search system to obtain a corresponding Chinese literature list as a search result.

[0040] This paper uses Baidu Academic as its academic literature retrieval system. Baidu Academic (https: / / xueshu.baidu.com) is a free academic resource search platform owned by Baidu. It covers a wide range of academic resources, including academic journals and conference papers, and aims to provide the best research experience for scholars at home and abroad. The Baidu Academic system is a comprehensive academic platform that integrates academic resource retrieval, academic services, and academic analysis, providing researchers with comprehensive, accurate, fast, and innovative academic resources and services.

[0041] Specifically, access the Baidu Academic system and enter the category name (obtained in step S11) as the search keyword into the Baidu Academic search box. The system will return a list of documents related to the search keyword. Browse the search results, paying attention to key information such as the title, author, and abstract to determine whether the retrieved documents are relevant to the data category to be filled.

[0042] Step S13: Use Selenium-based program code to automatically crawl the titles, abstracts, and keyword data of the Chinese literature list in the search results obtained in step S12 to obtain the literature data that needs to be filled in the data category.

[0043] Selenium is a tool for automated testing of web applications. It can run directly in the browser and simulate user operations in the browser, such as clicking, inputting, scrolling, etc., thereby realizing automated access to web pages and data crawling.

[0044] Specifically, the present invention realizes the automatic crawling of data by writing a Selenium script. In the Selenium script, WebDriver (a tool for automated testing of web applications) is utilized to access the page of the search results determined in step S12. Code is written to locate and extract the title, abstract, and keyword elements in the document list. Loop logic is written to traverse each item in the document list and crawl its title, abstract, and keyword data. The crawled data is stored in a text file with a suffix of .xsv. In this way, the document data that needs to be filled with data categories is obtained.

[0045] Step S14: Filter the document data that needs to be filled with data categories crawled in step S13, and use the filtered data to fill the data in the original Chinese document data set according to category to obtain a filled Chinese document classification data set.

[0046] Specifically, select the documents with complete content information and clear category information from the document data of the data categories to be filled crawled in step S13. According to the experience of predecessors in constructing incremental data sets, in this invention, the number of documents in each document category is fixed at 600. Then, use the filtered document data to fill the original Chinese document data set by category. As Figure 4 shown, this figure shows the data situation of the categories that need to be filled with data. Randomly select 600 document data from the categories in the original Chinese document data set whose number exceeds 600, and ensure that the total number of each category is 600. Therefore, the data set for Chinese document classification constructed in this invention has 20 document categories, and each category contains 600 data.

[0047] Step S15: Perform operations of removing spaces and stop words on the Chinese document data in the filled data set obtained in step S14 to obtain a benchmark data set for Chinese document classification problems.

[0048] Removing redundant spaces in the text (including leading and trailing spaces, unnecessary spaces in the middle, etc.) is an important step in data cleaning. These spaces may be generated during data entry or format conversion. They do not carry any useful information but will increase the storage space and processing time of the data. Stop words are words that frequently appear in the text but lack practical meaning, such as "de", "le", "he", etc. These words do not contribute much to the meaning analysis of the text but will occupy a large amount of storage space and computing resources. By removing stop words, the key words in the text can be made more prominent, which helps to extract and understand the theme and key information of the text. This is particularly important for literature data analysis because literature data usually contains a large number of technical terms and key information. For example, there is a Chinese document with the title information "The Core Issues of Vocational Education Teaching Reform in the New Era". The result after removing spaces and stop words from this title information is: "The Core Issues of Vocational Education Teaching Reform in the New Era".

[0049] The method for Chinese document classification based on incremental learning described in this invention further includes the following steps:

[0050] Step S2: Construct a benchmark model for Chinese document classification based on incremental learning.

[0051] As Figure 5 shown, in this invention, the benchmark model of the method for Chinese document classification based on incremental learning consists of six parts: an encoding layer, a convolutional layer, an activation layer, a max pooling layer, an attention layer, and a fully connected layer. The specific construction method is as follows:

[0052] Step S2 specifically includes:

[0053] Step S21: Set the encoding layer and use the BERT-wwm-ext, Chinese pre-trained language model to obtain the embedded representation of Chinese literature text data as the input of the subsequent model.

[0054] BERT-wwm-ext, Chinese is a Chinese pre-trained language model released by the Harbin Institute of Technology iFlytek Joint Laboratory (HFL). It is an upgraded version of BERT-wwm. By increasing the pre-training data set and the number of training steps and adopting full-word masking technology and Chinese word segmentation tools, the model performs well in Chinese NLP tasks. Its wide range of application scenarios and significant advantages make it an important technological achievement in the field of Chinese NLP. By using the BERT-wwm-ext, Chinese model, the present invention can convert Chinese document text data into a low-dimensional, dense numerical vector representation, which is called an embedding representation. This embedding representation can capture the semantic information in the text, enabling the model to better understand the text content.

[0055] Specifically, step S21 includes:

[0056] Step S211: Acquire Chinese document data in the benchmark dataset.

[0057] Step S212: Input the Chinese document data into the BERT-wwm-ext, Chinese pre-trained language model, and obtain the hidden state of the last layer in the model as the embedded representation vector of the Chinese document.

[0058] A Chinese document sequence is denoted as S. The Chinese document is input into the BERT-wwm-ext, Chinese pre-trained language model. The model uses the word segmentation algorithm to segment the text. The segmentation result is denoted as g(S). The formula is as follows:

[0059] g(S)={a1, a2, a3,…,a m ,…,a L};

[0060] Among them, a m This represents the mth token after word segmentation (the smallest unit into which text data is segmented during model processing). m is a subscript, meaning the number of tokens. L represents the number of tokens, which is 100 and is controlled by the max_length and padding parameters in the model.

[0061] In this invention, set max_length to 100 and padding to True, which means that regardless of the length of the original text, it will be truncated or padded (depending on the original length of the text) to ensure that the length of the final input sequence is 100. For how to perform word segmentation, for example, for the sequence "Core Issues in the Teaching Reform of Vocational Education in the New Era", the word segmentation results are: "new", "era", "vocational", "education", "teaching", "reform", "core", "issues".

[0062] After that, obtain the hidden state of the last layer in the model as the embedding representation vector of the Chinese literature, and denote the embedding representation vector of a Chinese literature as f(S). The formula is as follows:

[0063] f(S) = {e1, e2, e3, …, e m , …, e L};

[0064] Among them, e m represents the embedding representation corresponding to the m-th token, and its dimension is 768, which is set by the hidden_size parameter in the model. In BERT-wwm-ext, Chinese, hidden_size is set to 768, which means that the output dimension of the hidden layer is 768. L represents the number of tokens. e L represents the embedding representation corresponding to the L-th token. For example, in the sequence "Core Issues in the Teaching Reform of Vocational Education in the New Era", the embedding representation vector of "education" is [0.16714, -0.33311, -0.03177, 0.36868, -0.20583,......, 0.03909, -0.43855, 0.01807], and its dimension is 768.

[0065] Step S22: Set up a convolutional layer, and use convolutional kernels of different sizes to slide on the text vector to capture local features of different lengths.

[0066] Specifically, the text data is a one-dimensional sequence, so the number of input channels of the convolutional layer is 1, and at the same time, set the number of output channels of the convolutional feature map to 100. In this invention, the convolutional layer slides on the text embedding matrix through different convolutional kernels (also called filters) to extract local features of different lengths. For text data, the convolutional kernel can be regarded as a series of weights, which slide on the text and perform weighted summation on the data within each window to generate new features. Denote the output after convolution as y(x). The formula is as follows:

[0067] y(x) = A * x + b;

[0068] Among them, x is the text embedding matrix, whose dimensions are (batchsize, max_length, hidden_size), batchsize is the batch size, that is, the number of document samples contained in each batch, which is set to 128. max_length is the sequence length of 100, and hidden_size is the hidden layer output dimension size of 768. A is the weight of the convolution kernel, and its dimensions are defined as (h, d). h is the size of the convolution kernel, which defines the number of words covered by the convolution kernel. d is the same as the dimension of the text embedding, which ensures that the convolution kernel can be fully matched with the corresponding word vector in the text embedding matrix when sliding, thereby performing a convolution operation. b is the bias term. In the formula of the convolution operation, the multiplication involved in the symbol * is not the traditional matrix multiplication, but the process of multiplying the corresponding positions and then summing them.

[0069] Step S23: Set the activation layer and use the ReLU activation function to activate the features obtained after convolution.

[0070] Activation functions play a crucial role in neural networks. Their main role is to introduce nonlinear factors, improve the expressive power of the model, and enable neural networks to learn and simulate complex nonlinear relationships.

[0071] In this invention, the ReLU function is used as the activation function. The ReLU function can alleviate the gradient vanishing problem and speed up the training of the model. The result after the ReLU activation function is recorded as R(x), and the formula is as follows:

[0072] R(x)=max(0,y(x));

[0073] When the convolution value y(x) is greater than 0, the value itself is output; when the convolution value y(x) is less than or equal to 0, the output is 0. x is the text embedding matrix.

[0074] Step S24: Define the maximum pooling layer, perform dimensionality reduction on the features, and extract the most important features.

[0075] The max pooling layer extracts the maximum value within each pooling window as the representative feature of that window, effectively extracting the most significant and representative feature information in the text and achieving dimensionality reduction. This helps reduce the model's computational workload and number of parameters, improving the model's training speed and generalization ability. The result after max pooling is denoted as M(x), and the formula is as follows:

[0076] M(x) = maxpooling(R(x));

[0077] Among them, M(x) means extracting the most important features from the feature map generated by each convolution kernel, R(x) is the result after the ReLU activation function, and x is the text embedding matrix.

[0078] Step S25: Set the attention layer to obtain the weight of each feature and then multiply it with the original feature.

[0079] The attention mechanism can give different weights to each feature in the maximum pooling result, so that the features that are more critical to the classification task receive higher attention, thereby improving the classification performance of the model. In the present invention, the attention weight is calculated by a linear layer and a softmax function. Let the weight of the linear layer be R and the bias be θ. The weight and bias are randomly initialized during implementation. The calculated attention weight is recorded as η, and the formula is as follows:

[0080] η=σ(B·M(x)+θ);

[0081] Where σ is the softmax activation function, which is used to limit the output between 0 and 1 as the weight of each feature. M(x) is the result after maximum pooling, and x is the text embedding matrix.

[0082] Afterwards, the attention weight is multiplied by the output feature after the maximum pooling obtained in step S24 to obtain the weighted feature denoted as Q(x), which is as follows:

[0083] Q(x)=η⊙M(x);

[0084] Here, the symbol ⊙ represents element-wise multiplication. η is the attention weight, M(x) is the result after max pooling, and x is the text embedding matrix.

[0085] For example, the feature vector of the sequence "Core issues of vocational education and teaching reform in the new era" after passing through the encoding layer, convolution layer, activation layer, and maximum pooling layer is: [0.213, 0.324, 0.425, -0.736, 0.167, ..., 0.847, -0.625, 0.789], with a dimension of 100. Inputting this feature into the attention layer, the resulting attention weight vector is: [0.013, 0.015, 0.023, 0.004, 0.011, ..., 0.045, 0.005, 0.037], with a dimension of 100, and the sum of all elements is 1. The attention weight vector is multiplied by the corresponding position of the feature vector after the maximum pooling layer to obtain the weighted result: [0.002769, 0.00486, 0.009775, -0.002944, 0.001837, ..., 0.038115, -0.003125, 0.029193], whose dimension is 100.

[0086] Step S26: Define a fully connected layer to obtain the classification results of the model.

[0087] In classification tasks, the fully connected layer is responsible for mapping global features to specific category spaces and outputting prediction results for each category. This is achieved by calculating the linear combination of input features and the weight matrix.

[0088] In the present invention, three convolution functions with different convolution kernels are defined. The Chinese literature embedding matrix x is subjected to three convolution functions with different convolution kernels respectively, and then the three different convolution layer results are respectively passed through the activation layer, the maximum pooling layer and the attention layer in turn, and the obtained attention layer results are recorded as Q1(x), Q2(x), and Q3(x). These three attention layer results are spliced together by column and recorded as φ(x). Finally, φ(x) is input into the fully connected layer to obtain the prediction results for each category. Specifically, in the present invention, the bias term of the fully connected layer function is set to 0 in order to better apply the weight alignment method mentioned later. The output of the fully connected layer is recorded as o(x), and the formula is as follows:

[0089] o(x)=W T φ(x);

[0090] Among them, the dimension of o(x) is (batchsize, classes_num), classes_num is the number of output categories, which is 20. x is the text embedding matrix, W is the weight matrix of the fully connected layer, whose size is related to the dimension of the input feature vector and the number of output categories, T represents the transpose, W T Represents the transpose of the weight matrix. Each element in the weight matrix represents the strength of the association between the input feature and the output category.

[0091] The Chinese document classification method based on incremental learning of the present invention further includes the following steps:

[0092] In step S3, a flock selection algorithm is used to select representative samples from the old category and merge them with the new category data to construct an incremental learning dataset.

[0093] In incremental learning, the model retains previously learned knowledge, which helps it better maintain its adaptability to previous tasks. Incremental learning only involves updating new data, without retraining the entire model. This saves storage space and enables the model to utilize computing resources more efficiently. This invention employs an incremental learning approach, mixing representative data from old document categories with data from new document categories and incorporating them into the model for training. This approach effectively utilizes old data while ensuring the model adapts to new data.

[0094] Step S3 may specifically include:

[0095] Step S31: Use the flock selection algorithm to select representative training data from the old category.

[0096] The herding selection algorithm is a data screening method based on group behavior, which selects representative training data by simulating or utilizing the herding effect. Specifically, in the training data of each old category, a sorted list of samples of the category is generated based on the distance between the sample and the mean sample of the category to which it belongs. In this sorted sample list, the first u samples in the list are selected. These samples are most representative of the category to which they belong based on the mean. In the present invention, a sample set is set to store the data of the old category, and the size is fixed to 2000. The current number of old categories is recorded as v, and the following relationship exists:

[0097]

[0098] in, is a floor symbol, and u represents the number of samples selected from each old category for partial training data.

[0099] Step S32: The representative training data of the old category obtained in step S31 together with all the training data of the new category constitute a training data set for incremental learning.

[0100] The Chinese document classification method based on incremental learning of the present invention further includes the following steps:

[0101] In step S4, the incremental learning dataset is input into the benchmark model and trained under the constraints of the incremental learning loss function.

[0102] Specifically, the incremental learning dataset is input into the baseline model, and the model is trained under the constraints of the incremental learning loss function, which includes: the cross entropy loss function with the true label and the decoupled distillation loss function between the old model output and the new model output.

[0103] In the present invention, the fully connected layer output of a training sample is recorded as z, and the formula is:

[0104] z=[z1,z2,z3,…,z i ,…,z C ];

[0105] Among them, z i Represents the value corresponding to the i-th category in the output of the fully connected layer, and C is the number of categories.

[0106] z i The value after the softmax function is recorded as p i, which means the probability of the output being the i-th class, the formula is:

[0107]

[0108] Among them, exp( ) is the exponential function, exp(z i ) represents the z of e i j represents the data category, and its value range is 1 to C.

[0109] When the category label of a sample is t, the probability that the sample is output as t after being input into the model is called the probability of the target class, denoted as p t ; The probability of its output being other categories instead of t is called the probability of the non-target class, denoted as p \t .p t and p \t The formula is as follows:

[0110]

[0111] Among them, z t Represents the value corresponding to the target class in the fully connected layer output. k represents the data category, and k represents the remaining categories except the target class. j represents the data category, and its value range is 1 to C. k Represents the value corresponding to the kth class in the output of the fully connected layer, z j Represents the value corresponding to the jth class in the output of the fully connected layer.

[0112] In order to model the probability independently between non-target classes (not considering t classes), the present invention defines The formula is as follows:

[0113]

[0114] Among them, the value range of j is j∈{1, 2,…, t-1, t+1,…, C}, that is, j is the remaining categories except the target class.

[0115] The distillation loss formula used in the present invention is the decoupled distillation loss, denoted as l DKD , the formula is:

[0116] l DKD =αl TCKD +βl NCKD ;

[0117] Among them, α and β are the coefficients after decoupling, which are used to balance l TCKD and l NCKD The importance of TCKD It is called the target class knowledge distillation loss, which represents the similarity between the binary probabilities of the new and old models in the target class. NCKDIt is called the non-target class knowledge distillation loss, which represents the similarity between the probabilities of the new and old models in the non-target classes. TCKD and l NCKD The specific formula is as follows:

[0118]

[0119] in, and Representing the old model and the new model respectively. represents the probability that the fully connected layer outputs the target class in the old model, Represents the probability that the fully connected layer output is the target class in the new model. Indicates the probability that the fully connected layer outputs a non-target class in the old model, Represents the probability that the fully connected layer output is a non-target class in the new model. Indicates that the Indicates that the new model is calculated

[0120] In the present invention, the incremental learning loss function used in the training model includes the decoupled distillation loss and cross entropy loss. The incremental learning loss function is denoted as l, and the formula is:

[0121] l=λl DKD +(1-λ)l CE ;

[0122] Among them, λ is the coefficient used to balance the decoupled distillation loss and cross entropy loss. DKD is the decoupled distillation loss, l CE is the cross entropy loss, and the formula is:

[0123]

[0124] Among them, δ t＝i is the indicator function, p i is the probability that the output of the fully connected layer is the i-th category.

[0125] The Chinese document classification method based on incremental learning of the present invention further includes the following steps:

[0126] In step S5, the weight alignment method is used to update the weight of the new category output by the fully connected layer in the baseline model.

[0127] like Figure 6 As shown, the method used in the present invention consists of two stages. In the first stage ( Figure 6Left part), the present invention trains a new model on new data and representative old data, where the loss function is a combined loss including the decoupled distillation loss l DKD and cross entropy loss l CE In the second phase ( Figure 6 The present invention uses a weight alignment method to correct the bias weights in the training model. Represent the fully connected layer outputs of the current new model and the old model, o corrected Represents the fully connected layer output after correction using the weighted approach (WA).

[0128] Step S5 specifically includes the following steps:

[0129] Step S51: Calculate the norms of the old category and the new category weights in the fully connected layer weights respectively.

[0130] In the present invention, the fully connected layer weight W is defined as the following formula:

[0131] W=(W old , W new );

[0132] Among them, W old and W new Represent the weight of the old category and the weight of the new category in the fully connected layer weight. old and W new It can be expressed as the following formula:

[0133]

[0134] Among them, C old Represents the number of old categories, C new Represents the number of newly added categories. {1, 2, ..., C old} indicates the old category, {C old +1, C old +2,…,C old +C new} represents a new category. w1 refers to the weight of the first category in the fully connected layer weight, w2 refers to the weight of the second category in the fully connected layer weight, Refers to the weight of the Cth in the fully connected layer old The weight of each category, Refers to the weight of the Cth in the fully connected layer old +1 category weight, Refers to the weight of the Cth in the fully connected layer old +C new The weight of each category.

[0135] In this invention, the norm of the old category weight in the fully connected layer weight is recorded as Norm old . The norm of the new category weight in the fully connected layer weight is recorded as Norm new . Norm old and Norm new The formula is as follows:

[0136]

[0137] Among them, || || represents the norm operation. Refers to the weight of the Cth in the fully connected layer old The weight of each category is the norm, Refers to the weight of the Cth in the fully connected layer old +1 category weights take norm, Refers to the weight of the Cth in the fully connected layer old +C new The weight of each category is the norm.

[0138] Step S52: performing an averaging operation on the norms obtained in step S51.

[0139] In the present invention, the value after averaging the norm of the old category weights in the fully connected layer weights is recorded as Mean (Norm old ). The average value of the norm of the new category weight in the fully connected layer weight is recorded as Mean (Norm new ).

[0140] Step S53: Update the new category weights in the fully connected layer weights.

[0141] In the present invention, the Mean (Norm old ) and Mean(Norm new ) is recorded as γ, and the formula is as follows:

[0142]

[0143] The updated new category weight in the fully connected layer weight is recorded as The formula is as follows:

[0144]

[0145] Among them, W new is the weight of the new category in the fully connected layer weight.

[0146] In the present invention, the original fully connected layer output o(x) of the trained model can also be expressed as:

[0147]

[0148] Among them, old (x) indicates that the original fully connected layer outputs the value of the old category, o new (x) represents the value of the new category output by the original fully connected layer. and Represent the transpose of the old and new category weights in the fully connected layer, respectively. φ(x) is the column-wise concatenation of the attention layer results obtained by applying the Chinese document embedding matrix x to three convolution functions with different kernels, followed by the activation layer, the max pooling layer, and the attention layer.

[0149] The corrected fully connected layer output is recorded as o corrected (x), the formula is as follows:

[0150]

[0151] in, is the updated new category weight in the fully connected layer weight. As can be seen from the above formula, the final effect of the weight alignment method is to rescale the fully connected layer output to the value of the new category through the ratio coefficient γ.

[0152] The Chinese document classification method based on incremental learning of the present invention further includes the following steps:

[0153] Step S6: Input the Chinese document to be classified into the model after the weight alignment of the fully connected layer, perform sequence information recognition, and obtain the document classification result.

[0154] For example, "The core issues of vocational education and teaching reform in the new era. Since the new era, my country's vocational education reform has entered a critical stage of improving its connotation. In the overall picture of vocational education reform,..." is a Chinese document of unknown category. This document is input into the model as the data to be classified. It passes through the encoding layer, convolution layer, activation layer, maximum pooling layer, attention layer and fully connected layer in sequence, and the output category is "politics". Subsequently, the weight alignment method is applied, and this document is input into the model after weight alignment of the fully connected layer. At this time, the output document category is "education". After verification, the document does belong to the education category. Therefore, it can be seen that the model successfully identified the category of this Chinese document sequence, which reflects the effectiveness of the weight alignment method in correcting bias.

[0155] The beneficial effects of the present invention are:

[0156] This invention uses existing datasets to crawl data from the Baidu Academic system as a reliable data source, employing strict filtering conditions to ensure data accuracy. It also uses a pre-trained Chinese model to obtain embedded representations of Chinese document text sequences, effectively capturing the semantic information within the text. Furthermore, it applies incremental learning methods to the field of Chinese document classification, employing two approaches to address "catastrophic forgetting" and combining them with a TextCNN network model to identify Chinese document sequence information, resulting in more accurate and reliable classification results. Therefore, this invention offers the advantages of a novel approach and precise results.

[0157] In the present invention, data is crawled from the Baidu Academic system on the basis of an existing data set, and the data is subjected to strict pre-processing operations such as screening to construct a benchmark data set for the Chinese literature classification problem. The Chinese literature data set used in the present invention is a valuable resource contributed by Professor Li Ronglu of the Natural Language Processing Group of the International Database Center of the School of Computer Science and Technology of Fudan University. Baidu Academic is a powerful and resource-rich academic resource search platform that provides comprehensive, convenient and efficient academic services for scientific researchers. The present invention obtains data from the Baidu Academic system, and these data lay the foundation for the study of literature classification. The literature data obtained using crawler technology is merged with the literature data in the Fudan University Chinese Text Classification Dataset by category, and then the number of statistics and strict screening are performed. In short, through these pre-processing operations, a high-quality benchmark data set for the Chinese literature classification problem is constructed.

[0158] This paper designs an efficient baseline model for Chinese literature classification based on incremental learning. The model, consisting of an encoding layer, a convolutional layer, an activation layer, a maximum pooling layer, an attention layer, and a fully connected layer, can comprehensively and systematically identify the sequence information of Chinese literature. In particular, the present invention innovatively introduces an attention layer into the model, which assigns different weights to each feature after maximum pooling, ensuring that key features receive greater attention, thereby significantly improving the model's classification performance.

[0159] In this paper, a decoupled distillation loss combined with a cross-entropy loss is used to train the model. By adjusting the parameters of the two decoupled components of the original distillation loss, the model can better retain old knowledge while learning new knowledge. This paper also introduces a weight alignment method to overcome "catastrophic forgetting." The weights of the fully connected layer's output for new categories are updated, while the weights of the fully connected layer's output for old categories remain unchanged, resulting in a classification model with high accuracy and reliability.

[0160] The incremental learning-based Chinese literature classification method proposed in this paper uses a herd selection algorithm to select representative data from existing categories. Learning is then performed on the new category data along with representative data from existing categories. The incremental learning model learns knowledge under the constraints of an incremental learning loss function, not only learning new category information but also maintaining classification performance on test data from existing categories. This results in a more accurate, efficient, and low-memory Chinese literature classification method.

[0161] The present invention can adapt to the ever-increasing and changing nature of literature data, optimizing classification models through incremental learning of new data and improving classification accuracy and efficiency. This is of great significance for rapidly identifying and protecting intellectual achievements and promoting the effective use of literature information. It should be understood that the general description and detailed descriptions in this application are merely exemplary and explanatory and do not limit this application.

Claims

1. A Chinese literature classification method based on incremental learning, characterized in that: include: Step S1, constructing a benchmark dataset for Chinese document classification problem; Step S2, constructing a benchmark model for Chinese literature classification based on incremental learning; The benchmark model in step S2 is composed of a coding layer, a convolutional layer, an activation layer, a maximum pooling layer, an attention layer, and a fully connected layer connected in sequence, specifically including: Step S21: Set up the encoding layer and use the BERT-wwm-ext, Chinese pre-trained language model to obtain the embedded representation of Chinese literature text data as the input of the subsequent model; Step S22: Setting a convolution layer and sliding convolution kernels of different sizes on the text vector to capture local features of different lengths; Step S23: Set the activation layer and use the ReLU activation function to activate the features obtained after convolution; Step S24: define a maximum pooling layer to reduce the dimensionality of features and extract the most important features; Step S25: Set the attention layer to obtain the weight of each feature and then multiply it with the original feature; Step S26: define a fully connected layer to obtain the classification results of the model; Step S3: Use the flock selection algorithm to select representative samples from the old category and merge them with the new category data to construct an incremental learning dataset; Step S4: input the incremental learning dataset into the benchmark model and perform training under the constraints of the incremental learning loss function; Step S5, using the weight alignment method to update the weight of the new category output by the fully connected layer in the baseline model; Step S6: Input the Chinese document to be classified into the model after the weight alignment of the fully connected layer, perform sequence information recognition, and obtain the document classification result.

2. A Chinese document classification method based on incremental learning as claimed in claim 1, characterized in that: The step S1 specifically includes: Step S11: Download the Chinese literature classification data set and statistically analyze the data to determine whether data needs to be filled and determine the categories of data that need to be filled; Step S12: Using the category of data to be filled obtained in step S11 as a search condition, searching in an academic literature search system to obtain a corresponding Chinese literature list as a search result; Step S13: Use Selenium-based program code to automatically crawl the titles, abstracts, and keyword data of the Chinese literature list in the search results obtained in step S12 to obtain the literature data that needs to be filled in the data category; Step S14: screening the literature data that needs to be filled with data categories crawled in step S13, and using the screened data to fill the data in the original Chinese literature dataset by category to obtain a filled Chinese literature classification dataset; Step S15: performing operations of removing spaces and stop words on the Chinese document data in the filled data set obtained in step S14 to obtain a benchmark data set for the Chinese document classification problem.

3. The Chinese document classification method based on incremental learning according to claim 1, characterized in that: The step S21 specifically includes: Step S211: Acquire Chinese document data in the benchmark dataset; Step S212: Input the Chinese document data into the BERT-wwm-ext, Chinese pre-trained language model, and obtain the hidden state of the last layer in the model as the embedded representation vector of the Chinese document.

4. A Chinese document classification method based on incremental learning as claimed in claim 3, characterized in that: In step S212: A Chinese document sequence is denoted as S. The Chinese document is input into the BERT-wwm-ext, Chinese pre-trained language model. The model uses the word segmentation algorithm to segment the text. The result after word segmentation is denoted as g(S). The formula is as follows: <h2 style=";text-align:left;direction:ltr">g(S) = {a1,a2,a3,...,a<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr"> ,...,a<h2 style=";text-align:left;direction:ltr"> L <h2 style=";text-align:left;direction:ltr">}; Among them, a m Indicates the mth token after word segmentation, m represents the subscript, meaning the number; L represents the number of tokens, which is 100 and is controlled by the max_length and padding parameters in the baseline model. Then, the hidden state of the last layer in the baseline model is obtained as the embedded representation vector of the Chinese document. The embedded representation vector of a Chinese document is denoted as f(S). The formula is as follows: f(S)={e1,e2,e3,...,e m ,...,e L }; Among them, e m represents the embedding representation corresponding to the mth token, whose dimension is 768, which is set by the hidden_size parameter in the baseline model. In BERT-wwm-ext, Chinese, hidden_size is set to 768, which means that the hidden layer output dimension size is 768. L represents the number of tokens, e L Represents the embedding representation corresponding to the Lth token.

5. The Chinese document classification method based on incremental learning according to claim 4, characterized in that: The step S22 specifically includes: For text data, the convolution kernel can be regarded as a series of weights that slide over the text and perform weighted summation on the data in each window to generate new features. The output after convolution is recorded as y(x), and the formula is as follows: y(x)=A*x+b; Where x is the text embedding matrix, whose dimension is (batchsize, max_length, hidden_size), batchsize is the batch size, which is set to 128; max_length is the sequence length of 100, hidden_size is the hidden layer output dimension size of 768; A is the weight of the convolution kernel, and its dimension is defined as (h, d); h is the size of the convolution kernel, which defines the number of words covered by the convolution kernel; d is the same dimension as the text embedding; b is the bias term; in the convolution operation formula, the multiplication involved by the symbol * is the process of multiplying the corresponding positions and then summing them.

6. A Chinese document classification method based on incremental learning as claimed in claim 5, characterized in that: The step S23 specifically includes: Use the ReLU function as the activation function and record the result after the ReLU activation function as R(x). The formula is as follows: R(x)=max(0,y(x)); Among them, when the convolved value y(x) is greater than 0, the value itself is output; when the convolved value y(x) is less than or equal to 0, the output is 0; x is the text embedding matrix.

7. A Chinese document classification method based on incremental learning as claimed in claim 6, characterized in that: The step S24 specifically includes: The result after maximum pooling is recorded as M(x), and the formula is as follows: M(x) = maxpooling(R(x)); Among them, M(x) means extracting the most important features from the feature map generated by each convolution kernel, R(x) is the result after the ReLU activation function, and x is the text embedding matrix.

8. The Chinese document classification method based on incremental learning according to claim 1, characterized in that: The step S3 specifically includes: Step S31: Use the flock selection algorithm to select representative training data from the old category; Step S32: The representative training data of the old category obtained in step S31 together with all the training data of the new category constitute a training data set for incremental learning.

9. The Chinese document classification method based on incremental learning according to claim 1, characterized in that: The step S5 specifically includes: Step S51: Calculate the norms of the old category and the new category weights in the fully connected layer weights respectively; Step S52: performing an averaging operation on the norms obtained in step S51; Step S53: Update the new category weights in the fully connected layer weights.

Images