Layered topic classification continuous training method capable of dynamically expanding categories
By constructing historical data in hierarchical topic classification, using fallback data and existing models to mix training, the model forgetting problem when label structure changes is solved, and efficient dynamic expansion and learning ability improvement is achieved.
Patent Information
- Application Number
- CN202411518337.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-08-12
AI Technical Summary
The existing hierarchical topic classification methods cannot efficiently expand dynamically when the labels and their structures change, resulting in the model forgetting the original knowledge and retraining is expensive or impossible.
Construct historical data through fallback operations, and use existing models and fallback data to perform continuous training to adapt to changes in label structure and avoid model forgetting.
Without retraining the model, use a small amount of new data to simulate historical data, maintain the consistency of model learning, improve the ability to dynamically expand the label structure, and reduce resource waste.
Smart Images
Figure CN120470448A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing, and in particular relates to a neural network method for continuous learning of dynamically expanded hierarchical topic classification. Background Art
[0002] Hierarchical topic classification refers to a multi-label classification method in which the classification labels have a tree-like hierarchical structure. Many common classification scenarios have hierarchical structures, such as the four-level classification structure of legal cases. For a piece of input text, hierarchical topic classification requires the model to generate classification labels from the highest level to the lowest level within a given label structure.
[0003] Existing hierarchical topic classification methods treat the classification labels and their tree-like hierarchical structure as fixed, and the classifier is trained based on this fixed structure. Each classification is also performed based on the fixed labels and hierarchy.
[0004] However, in real-world applications, labels and their structures are not always fixed. As things evolve, new class labels may need to be introduced, requiring the label tree to be dynamically expanded. Retraining a model from scratch to address changes in labels and their structures would be extremely time-consuming and resource-intensive, and retraining might even be impossible due to loss of the original training data or security concerns. This requires the model to be able to continuously train the existing classification model for the newly added labels using a small amount of data related to the new labels. However, simply training an existing model with new data can easily cause the model to forget its previously acquired knowledge, affecting its ability to classify the original type of data and ultimately affecting overall performance. Summary of the Invention
[0005] The present invention provides a hierarchical topic classification continuous training method with dynamically expandable categories, which is used to improve the topic classification effect of the model during the expansion of the classification label structure.
[0006] The technical solutions provided by the present invention are as follows:
[0007] A method for continuous training of hierarchical topic classification with dynamically scalable categories, comprising the following steps:
[0008] A. Continuous training for hierarchical topic classification, with the option to roll back to a historical stage or not based on label expansion data;
[0009] B. If you choose to roll back to a certain historical stage, perform a rollback operation on the data of that historical stage. Specifically, the rollback operation involves calculating the difference between the current label structure and the target label structure to be rolled back. For the target label set of the selected data, change it to the difference between the target label set and the label structure difference, thereby constructing the rollback data;
[0010] C. Use fallback data and historical data to continuously train the existing model.
[0011] Furthermore, the continuous training in step A has two or more stages. When the label is expanded, the current label structure is recorded for future label rollback operations.
[0012] Furthermore, the model in step A takes text and label structure as input. When the target label structure is expanded, it is continuously trained based on the existing model to adapt to the changes in the label structure.
[0013] Furthermore, in step A, the model uses a uniform distribution to randomly select from all historical stages with a certain probability.
[0014] Furthermore, in step C, the fallback data is mixed with possible historical data to form a training data set, and the current model is directly based on the fallback data to continue training.
[0015] Beneficial effects of the present invention:
[0016] The present invention only utilizes a small amount of data for label structure changes in the latest stage. By using a part of the data to simulate historical data, the neural network model can be prevented from forgetting. On the basis of the original model, only the data for new needs is used to adjust the model, rather than training the model from scratch. In addition, the fallback data is processed in advance before training, rather than being dynamically generated during the training process. Since the same data may be trained multiple times during neural network training, a pre-processing method is used to ensure that the target labels corresponding to the same text are the same each time training. Compared with the practice of dynamically generating label fallback results during the training process, it ensures that the answers to the same text do not change, which is more conducive to model learning. Moreover, the present invention does not require a specific neural network structure. For any neural network model used for hierarchical topic classification, the dynamic expansion of the label structure can be completed through the continuous training of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A flowchart of the method for continuous training of hierarchical topic classification with dynamically scalable categories according to the present invention;
[0018] Figure 2 Schematic diagram of the label rollback process of the present invention;
[0019] Figure 3 Schematic diagram of the operation of the label backoff continuous training method of the present invention. DETAILED DESCRIPTION
[0020] The present invention will be further described below by way of examples.
[0021] The present invention takes the following specific implementation as an example, in which the topic type tag structure is from T1, T2, ..., T t-1 Expanded to T t After that, given a batch of data D t =(X,Y), Neural network model needs to adapt to T t . The core of the present invention is to perform rollback for a small amount of label expansion data, and continuously train the neural network model using a mixture of historical data simulated by rollback and the latest data, to complete the dynamic expansion of the label structure on the hierarchical topic classification task. And retaining all historical stages can ensure that the model can always see the label structure of all stages during the continuous training process, thereby enhancing memory ability. At the same time, label rollback is performed before training to ensure that the label corresponding to the same text remains unchanged during the training process, avoiding confusion caused by label conflicts.
[0022] The present invention provides a hierarchical subject classification continuous training method with dynamically expandable categories. The specific process is as follows: Figure 1 As shown, including:
[0023] A. For continuous training of hierarchical topic classification, we choose to roll back to the historical stage or not roll back based on the label expansion data. That is, we select label rollback for the data in continuous training based on the historical label structure. This is to reduce the forgetting phenomenon of the neural network model when the label structure is dynamically expanded. Specifically, it includes:
[0024] A1. For continuous training of hierarchical topic classification, the latest hierarchical label structure is restored to the label structure before dynamic expansion. At the same time, for the true label corresponding to a text segment in the training data, the label that belongs to the difference between the latest label structure and the historical label structure is removed.
[0025] A2. The model encodes the text and label structure information and outputs a classification result based on this information. The model does not require a specific structure, but it must be able to accept text input and understand the structure of the label. After encoding these two components, the model generates a classification result by predicting the probability of all classes.
[0026] A3. Use a uniform distribution to select label rollback. For a small amount of continuous learning data with changing labels, randomly choose to roll back to any historical stage or not roll back according to a certain probability. That is, according to different needs, set the rollback probability p1, p2, ..., p for each stage. t-1 ,satisfy For a piece of data (X i ,Y i ), randomly select the stage to fall back to or not to fall back according to the fallback probability.
[0027] B. If you choose to roll back to a certain historical stage, perform a label rollback operation on that historical stage to obtain the rollback data. Figure 2 As shown in the figure, the label rollback operation is to calculate the difference between the current label structure and the target label structure to be rolled back, and change the target label set of the selected data to the difference between the target label set and the label structure difference. Specifically, for a piece of data (X, Y), the label structure selected for the rollback phase is T j , then construct new training data in The result of deleting the extended part of the label structure in Y. If you choose not to roll back, the original data remains unchanged. t Perform the above operations on all data in to get the fallback dataset
[0028] C. Use fallback data and possible historical data to continuously train based on the existing model, refer to Figure 3 , the model is continuously trained by back-off dataset. A hierarchical topic classification neural network model Θ needs to estimate the following conditional probabilities:
[0029] P(Y|X,T;Θ)
[0030] When predicting, if for a certain type y i There is P(y i |X,T;Θ)>δ, then the text X is considered to belong to this category, where δ is a pre-set threshold, usually set to 0.5.
[0031] At time t, there is a model Θ t-1 This can be done in the tag structure T t-1 The classification P(Y|X,T t-1 ;Θ t-1 ), need to be in data D t Continue training to adapt to the label structure expansion to T t Using the above backoff dataset Use maximum likelihood estimation for training. For n labels, initialize the neural network parameters Θ t =Θ t-1 , the training target is the following log-likelihood function:
[0032]
[0033] Although the training data is only for the latest label structure T t , the fallback technology additionally brings the historical structure T1,T2,…,T t-1 Information. Historical data D1, D2, ..., D t-1 Although not required, it is helpful to have some historical data. Can be used with fallback data Mixed, also use the above objective function training. If conditions permit, you can also retain part of the current data D t join in Facilitates future tag structure expansion.
[0034] The technology involved in the present invention is not limited to the above-mentioned problem of continuous training of hierarchical subject classification. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.
Claims
1. A method for continuous training of hierarchical topic classification with dynamically scalable categories, comprising the following steps: A. Continuous training for hierarchical topic classification, with the option to roll back to a historical stage or not based on label expansion data; B. If you choose to roll back to a certain historical stage, perform a rollback operation on the data of that historical stage. Specifically, the rollback operation involves calculating the difference between the current label structure and the target label structure to be rolled back. For the target label set of the selected data, change it to the difference between the target label set and the label structure difference, thereby constructing the rollback data; C. Use fallback data and historical data to continuously train the existing model.
2. The method for continuous training of hierarchical topic classification with dynamically scalable categories as claimed in claim 1, characterized in that: The continuous training in step A has two or more stages. When the label is expanded, the current label structure is recorded for future label rollback operations.
3. The method for continuous training of hierarchical topic classification with dynamically scalable categories as claimed in claim 1, characterized in that: In step A, the model takes text and label structure as input. When the target label structure is expanded, it continues to train based on the existing model to adapt to the changes in the label structure.
4. The method for continuous training of hierarchical topic classification with dynamically scalable categories as claimed in claim 1, characterized in that: In step A, the model uses a uniform distribution to randomly select from all historical stages with a certain probability.
5. The method for continuous training of hierarchical topic classification with dynamically scalable categories as claimed in claim 1, characterized in that: In step C, the fallback data is mixed with any existing historical data to form a training data set, and the current model is directly based on the fallback data to continue training.