A multi-stage training method and system of a multi-view-based text classification model

Abstract

The application provides a multi-stage training method and system of a multi-view-based text classification model, comprising: performing mask processing on original text data and performing preliminary training on a pre-training model based on the processed data; performing intermediate training in a second stage based on all text data and corresponding text word frequency pseudo labels; and performing third stage training based on preset proportion of text data and corresponding text topic view pseudo labels. The multi-stage training method and system of the multi-view-based text classification model can analyze text data from multiple angles, extract multiple view features including text word frequency view and text topic view, and generate pseudo labels based on these features. Subsequently, the pre-training model is subjected to multi-stage intermediate training. After completing the multi-stage intermediate training, the model is further fine-tuned using a small amount of labeled data, thereby enhancing the adaptability of the model in the target field.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multi-stage training method and system for a text classification model based on multiple views. Background Technology

[0002] Text classification is a core task in the field of natural language processing, aiming to categorize text into predetermined classes based on its content. With the rapid development of information technology and the surge in text data volume, how to efficiently and massively achieve rapid text classification has always been a pressing technical problem to be solved. Summary of the Invention

[0003] One objective of this invention is to provide a multi-stage training method and system for a multi-view text classification model. This method involves analyzing text data from multiple perspectives, extracting various view features including a word frequency view and a topic view, and generating pseudo-labels based on these features. Subsequently, the pre-trained model undergoes multi-stage intermediate training. After completing these intermediate training stages, the model is further fine-tuned using a small amount of labeled data, thereby enhancing its adaptability to the target domain and ensuring excellent robustness and good generalization ability even with scarce data. The model obtained through this training enables efficient and rapid classification of large-scale text data.

[0004] This invention provides a multi-stage training method for a text classification model based on multiple views, comprising:

[0005] The original text data is masked and the pre-trained model is initially trained based on the processed data.

[0006] The second stage of intermediate training is conducted based on all text data and their corresponding text word frequency pseudo-labels.

[0007] The third stage of training is conducted using text data of a preset proportion and its corresponding text topic view pseudo-labels.

[0008] Preferably, the multi-stage training method for a multi-view-based text classification model further includes:

[0009] Based on a preset amount of labeled data, the model is fine-tuned.

[0010] Preferably, the preliminary training steps are as follows:

[0011] The original text data is formatted and a probability matrix of masks is generated for each basic unit after conversion.

[0012] The difference between the model's predictions and the true labels is calculated using the sparse multi-class cross-entropy loss function; the model is trained using the Adam optimizer, the loss function is minimized using gradient descent, and the model weights are updated.

[0013] Preferably, during the intermediate training phase of the second stage, the cross-entropy loss function is selected to calculate the difference between the class predicted by the model and the pseudo-label, and the Adam optimizer is used for backpropagation to calculate the gradient and update the model parameters.

[0014] Preferably, the trained model is associated based on the correlation between the text data used in the third stage of training, and the associated model is called for auxiliary verification when the trained model is used for classification.

[0015] This invention also provides a multi-stage training system for a text classification model based on multiple views, comprising: a preliminary training module, a second-stage training module, and a third-stage training module; wherein, the preliminary training module performs masking processing on the original text data and performs preliminary training on the pre-trained model based on the processed data; the second-stage training module performs a second-stage intermediate training based on all text data and its corresponding text word frequency pseudo-labels; and the third-stage training module performs a third-stage training based on a preset proportion of text data and its corresponding text topic view pseudo-labels.

[0016] Preferably, the multi-stage training system for a text classification model based on multiple views further includes: a fine-tuning module for fine-tuning the model based on a preset amount of labeled data.

[0017] Preferably, the preliminary training steps are as follows:

[0018] The original text data is formatted and a probability matrix of masks is generated for each basic unit after conversion.

[0019] The difference between the model's predictions and the true labels is calculated using the sparse multi-class cross-entropy loss function; the model is trained using the Adam optimizer, the loss function is minimized using gradient descent, and the model weights are updated.

[0020] Preferably, during the intermediate training phase of the second stage, the cross-entropy loss function is selected to calculate the difference between the class predicted by the model and the pseudo-label, and the Adam optimizer is used for backpropagation to calculate the gradient and update the model parameters.

[0021] Preferably, the trained model is associated based on the correlation between the text data used in the third stage of training, and the associated model is called for auxiliary verification when the trained model is used for classification.

[0022] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0024] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0025] Figure 1 This is a schematic diagram of a multi-stage training method for a text classification model based on multiple views, as described in an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of a multi-stage training system for a text classification model based on multiple views, as described in an embodiment of the present invention. Detailed Implementation

[0027] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0028] This invention provides a multi-stage training method for a text classification model based on multiple views, such as... Figure 1 As shown, it includes:

[0029] Step 1: Mask the original text data and perform preliminary training on the pre-trained model based on the processed data;

[0030] The pre-training process of the pre-trained model is as follows: First, the text data is preprocessed, and the preprocessed data is used as the original text data. Preprocessing includes operations such as removing stop words and punctuation, stemming, and TF-IDF vectorization. Then, the preprocessed text data is clustered using sIB or K-means with a specified number of clusters to generate pseudo-labels for the text term frequency view. A large number of topics is set, and this number of topics is used to train the LDA model to infer the possible topics of each article. The topic distribution of each article is obtained. Based on the trained LDA model, the topic with the highest weight for each article is selected as the article's topic label, i.e., the pseudo-label for the text topic view.

[0031] Preprocessing of text data includes: reading all stop words from a file or a predefined stop word list; removing stop words and punctuation marks using regular expressions and string manipulations; and restoring each word to its root form. For each text, the term frequency of each word is calculated. In the formula n d `w` represents the total number of words in text `d`, and `Count(w,d)` represents the number of times word `w` appears in text `d`. Calculate inverse text frequency. In the formula, N is the number of texts, and DF(w) is the number of texts containing word w. For each word w in a text, calculate TF-IDF(w,d) = TF(w,d)·IDF(w); for each text, a maximum of 10,000 feature words are considered, and those exceeding this number are ignored, and a TF-IDF feature matrix is ​​constructed.

[0032] The steps for training an LDA model are as follows: Based on the constructed TF-IDF feature matrix, perform sIB or K-means clustering on the text data, dividing the text data into K clusters, each cluster being a pseudo-label of the text term frequency view; based on the constructed TF-IDF feature matrix, set a relatively large number of topics and use the LdaModel tool from the gensim library for LDA training. The training process is represented as follows: In the formula, m represents the generation of the m-th article, and O represents the generation of the O-th word. m Let represent the total number of words in the m-th article, α be the hyperparameter of the topic distribution, β be the hyperparameter of the vocabulary distribution, and θ be the probability of the topic distribution for a given article. Let R be the probability distribution of words under a certain topic, R be the possible topics that a word in an article might fit, and G be the words generated by R. The training process is essentially a process of randomly generating each word in the article in a probabilistic way. After the LDA model is trained, the topic distribution of each article is obtained. The topic distribution of each article is a probability distribution, representing the weight of the article on each topic. The topic with the highest probability in the topic distribution is selected as the article's topic label, i.e., the pseudo-label of the text topic view.

[0033] The initial training steps are as follows:

[0034] The original text data is format-converted, and a probability matrix for masking is generated for each converted basic unit. This involves converting the text data into token IDs in the input format of the pre-trained model, ensuring that the length of all text does not exceed 128 tokens (basic units). If the text length exceeds 128, it is truncated; if the text length is less than 128, it is padded. A probability matrix for masking is generated for each token. The total probability of the mask is set to 0.15, meaning 15% of the tokens will be masked. Of these 15% of masked tokens, 80% are replaced with the [MASK] token, 10% are randomly replaced with other tokens from the model vocabulary, and the remaining 10% remain unchanged.

[0035] The difference between the model's predictions and the true labels is calculated using the sparse multi-class cross-entropy loss function. The model is trained using the Adam optimizer, and the loss function is minimized via gradient descent to update the model's weights. Specifically, the difference between the model's predictions and the true labels is calculated using the sparse multi-class cross-entropy loss function, as shown in the formula: In the formula, H represents the number of samples in the batch, C represents the total number of categories, i.e., the size of the vocabulary, and z i,c y is the model's predicted score for the i-th sample in class C. i It is the true label of the i-th sample. This represents the model's predicted score for the true label of the i-th sample. The model is trained using the Adam optimizer, and the loss function is minimized using gradient descent to update the model's weights.

[0036] Step 2: Conduct the second stage of intermediate training based on all text data and their corresponding text word frequency pseudo-labels;

[0037] In the second stage of intermediate training, the cross-entropy loss function is used to calculate the difference between the model's predicted class and the pseudo-label. The Adam optimizer is then used for backpropagation to calculate the gradient and update the model parameters. Specifically, the preprocessed text data is converted into token IDs in the input format of the pre-trained model and input along with the corresponding word frequency view pseudo-labels into the pre-trained model. The cross-entropy loss function is used to calculate the difference between the model's predicted class and the pseudo-label, using the following formula: Where q i It is a pseudo-label of the sample, p i The prediction model calculates the probability, where L is the total number of training text data. Backpropagation is performed using the Adam optimizer to compute gradients and update model parameters.

[0038] Step 3: Conduct the third stage of training based on a preset proportion of text data and its corresponding text topic view pseudo-labels. This involves converting the pre-processed text data into token IDs required for the pre-trained model. A small proportion of the token IDs are randomly selected as training data, and these samples, along with their corresponding text topic view pseudo-labels, are input into the pre-trained model that has already undergone a second intermediate training. This allows the model to effectively learn the feature representations in the text topic views. The cross-entropy loss function is then used to calculate the difference between the model's predicted category and the pseudo-label, using the following formula: Where q i It is a pseudo-label of the sample, p i The prediction model calculates the probability, where L is the total number of training text data. Backpropagation is performed using the Adam optimizer to compute gradients and update the model parameters.

[0039] To further optimize the model for more accurate text data classification, in one embodiment, the multi-stage training method for the multi-view text classification model further includes: fine-tuning the model based on a preset amount of labeled data. This involves inputting a small amount of labeled data and corresponding real labels into the pre-trained model that has already undergone intermediate training in the third stage, performing backpropagation using the Adam optimizer, and optimizing the model using batch gradient descent or mini-batch gradient descent.

[0040] Since the amount of data used in the third stage of training directly affects the applicability of the model to the application scenario, in order to address this issue, in one embodiment, the trained model is associated based on the correlation between the text data used in the third stage of training, and the associated model is called for auxiliary verification when the trained model is used for classification.

[0041] After training, the model constructs a model invocation parameter set based on the classification numbers of the pre-configured domain classification tree corresponding to the data used in the third stage of training. This involves statistically analyzing the classifications corresponding to the domain classification tree used in the training, and using the statistical results to construct the model invocation parameter set. Each parameter in the model invocation parameter set corresponds to a category in one of the statistically analyzed domain classification trees. When applied to text data classification scenarios, accurate model invocation is crucial. To quickly invoke the model, a comprehensive analysis of the devices and personnel uploading the original text data is conducted. This involves analyzing the classification of the first historical upload records of devices and the classification of the second historical upload records of personnel, and constructing a corresponding model invocation priority table, prioritizing model invocation. The table uses a unique identifier code to call the model; a model call timeline is constructed by comprehensively analyzing the classification of the first and second historical upload records; a priority value is determined for each call time on the model call timeline using a pre-configured priority value determination table (the closer the call time is to the current time, the higher the priority value); the number of calls to each model and the call time corresponding to each call are counted for a preset number of times (any number from 10 to 1000); an analysis value is determined based on each call time and whether it belongs to the first or second historical upload record; the analysis value is the sum of the products of the priority value corresponding to each call time and the pre-configured coefficient determined by whether it belongs to the first or second historical upload record.

[0042] During model usage, it is essential to continuously ensure the accuracy and effectiveness of the model's classification, making model updates indispensable. However, frequent updates are not advisable as they can waste computer resources. The steps for determining the timing of updates are as follows: First, consider whether the model needs updating. Then, analyze the data operation on the platform and update the model when the platform is idle, balancing platform operation and model update needs. In one embodiment, multi-stage training of a multi-view text classification model further includes: quantifying the model's online time (the continuous running time since the last update) based on a preset first stability analysis table to determine a first stable value; analyzing the model's usage feedback based on a preset second stability analysis table to determine a second stable value; and analyzing the model's domain based on a preset third stability analysis table. The technological innovation situation is analyzed to determine the third stable value. The sum of the first, second, and third stable values ​​is used as the criterion. When the value is less than or equal to the preset stability threshold, the model is placed in the update requirement list to await update. In the first stability analysis table, the online time is associated with the first stable value. In the second stability analysis report, the second stable value is associated with the feedback parameters of the feedback situation. The feedback parameters include: parameters indicating whether negative feedback exists, parameters indicating the proportion of negative feedback, parameters indicating the maximum authority among those who issued negative feedback, and parameters indicating the number of those who issued negative feedback. In the third stability analysis table, the characterization parameters of the innovation situation are associated with the third stable value. The characterization parameters include: parameters indicating whether innovation has occurred and parameters indicating the impact of innovation.

[0043] When updating the model, the platform load and the model update requirements must be considered. First, based on a pre-trained platform load prediction model, the platform load is predicted to determine the remaining load. Based on the remaining load, models eligible for updating are selected from the update requirement list. If multiple models are selected, a second screening is performed, considering the model's usage frequency and the last update time. The second screening steps are as follows: A first screening value is determined based on a pre-configured usage frequency quantification table; a second screening value is determined based on a pre-configured update time quantification table; the update target is the one with the largest sum of the first and second screening values. Higher usage frequency results in a higher first screening value; earlier update time results in a higher second screening value. The platform load prediction model is a neural network model trained to predict future loads by comprehensively analyzing the platform's historical operating records.

[0044] To evaluate the effectiveness of the model update training process and indirectly verify that the update meets expectations, the following steps are taken: The data used in this update is compared with the data used in the previous update to determine the values ​​of each preset comparison parameter; a quantization value is determined based on the quantization scoring criteria configured for each comparison parameter; the sum of the products of the quantization values ​​and their corresponding weights is used as the performance score; if the performance score is greater than or equal to a preset threshold, the update evaluation is passed; otherwise, data is reselected for the update. Since this application employs a three-stage training process with potential fine-tuning, the evaluation needs to consider the specific characteristics of the model training. Performance scores are calculated for each stage, and then the three or four performance scores are weighted and summed to obtain the final score. The weights used for each weighting step are pre-configured by professionals. Furthermore, the threshold can be determined by querying a pre-configured threshold determination table based on the first stable value, the second stable value, and the third stable value.

[0045] This invention also provides a multi-stage training system for a text classification model based on multiple views, such as... Figure 2 As shown, it includes: a preliminary training module 1, a second-order training module 2, and a third-order training module 3; wherein, the preliminary training module 1 performs masking processing on the original text data and performs preliminary training on the pre-trained model based on the processed data; the second-order training module 2 performs the second-stage intermediate training based on all text data and its corresponding text word frequency pseudo-labels; the third-order training module 3 performs the third-stage training based on a preset proportion of text data and its corresponding text topic view pseudo-labels.

[0046] The multi-stage training system for the text classification model based on multiple views also includes a fine-tuning module, which is used to fine-tune the model based on a preset amount of labeled data.

[0047] The initial training steps are as follows:

[0048] The original text data is formatted and a probability matrix of masks is generated for each basic unit after conversion.

[0049] The difference between the model's predictions and the true labels is calculated using the sparse multi-class cross-entropy loss function; the model is trained using the Adam optimizer, the loss function is minimized using gradient descent, and the model weights are updated.

[0050] In the second stage of intermediate training, the cross-entropy loss function is selected to calculate the difference between the model's predicted class and the pseudo-label. The Adam optimizer is used for backpropagation to calculate the gradient and update the model parameters.

[0051] In this process, the trained models are associated based on the correlation between the text data used in the third stage of training, and the associated models are called for auxiliary verification when the trained models are used for classification.

[0052] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-stage training method for a text classification model based on multiple views, characterized in that, include: The text data is preprocessed and used as the original text data. The preprocessing includes removing stop words and punctuation marks, stemming, and TF-IDF vectorization. Based on the constructed TF-IDF feature matrix, the text data is clustered using sIB or K-means to divide the text data into multiple clusters, and each cluster is used as a pseudo-label for the text word frequency view. Based on the TF-IDF feature matrix, an LDA model is trained to obtain the topic distribution of each text, and the topic with the highest probability in the topic distribution is selected as the pseudo-label for the text topic view. The original text data is masked and the pre-trained model is initially trained based on the processed data. The second stage of intermediate training is conducted based on all text data and its corresponding text word frequency view pseudo-labels. The third stage of training is conducted based on randomly selected text data of a preset proportion and its corresponding text topic view pseudo-labels. For the trained model, based on the classification numbers of the pre-configured domain classification tree corresponding to the data used in the third stage of training, a model call parameter set is constructed. Specifically, the classification of the data used in the domain classification tree is statistically analyzed, and the model call parameter set is constructed based on the statistical results. Each parameter in the model call parameter set corresponds to a category of a domain classification tree. When applying the trained model to a text data classification scenario, the classification of the first historical upload records of the device that uploaded the original text data and the classification of the second historical upload records of the person who uploaded the original text data are comprehensively analyzed to construct a corresponding model call priority table, and the model call priority table is used for model calling. The model is called using a unique identifier code in the priority table; the comprehensive analysis includes: constructing a model call timeline based on the classification of the first and second historical upload records; determining the priority value corresponding to each call time on the model call timeline according to a pre-configured priority value determination table; counting the number of calls to each model and the call time corresponding to each call for a preset number of times; determining the analysis value based on each call time and whether the call belongs to the first or second historical upload record; the analysis value is the sum of the product of the priority value corresponding to each call time and the pre-configured coefficient determined based on whether the call belongs to the first or second historical upload record. During model usage, the model's online time is quantified based on a preset first stability analysis table to determine a first stable value; the model's usage feedback is analyzed based on a preset second stability analysis table to determine a second stable value; and the technological innovation in the model's field is analyzed based on a preset third stability analysis table to determine a third stable value. The sum of the first, second, and third stable values ​​is used as a criterion. When the sum is less than or equal to a preset stability threshold, the model is placed in an update requirement list awaiting an update. When updating the model, the platform load is predicted based on a pre-trained platform load prediction model to determine the remaining load. Models eligible for updating are then selected from the update request list based on the remaining load. When multiple models are selected, a first selection value is determined based on a pre-configured usage frequency quantization table, and a second selection value is determined based on a pre-configured update time quantization table. The model with the largest sum of the first and second selection values ​​is used as the update target. The higher the usage frequency, the higher the first selection value; the earlier the update time, the higher the second selection value. The platform load prediction model is a neural network model, trained to predict future load based on the platform's historical operating records. When updating the target, the data used in each of the initial training, the second intermediate training, and the third training stages are compared with the data used in the previous update to determine the values ​​of each preset comparison parameter. A quantized value is determined based on the quantization scoring standard configured for each comparison parameter, and the sum of the products of the quantized values ​​and their corresponding weights forms the performance score for each stage. The performance scores for the three stages are weighted and summed to obtain the final score. If the final score is greater than or equal to a preset threshold, the update evaluation is passed; otherwise, data is reselected for updating.

2. The multi-stage training method for a text classification model based on multiple views as described in claim 1, characterized in that, Also includes: Based on a preset amount of labeled data, the model is fine-tuned.

3. The multi-stage training method for a text classification model based on multiple views as described in claim 1, characterized in that, The initial training steps are as follows: The original text data is formatted and a probability matrix of masks is generated for each basic unit after conversion. The difference between the model's predictions and the true labels is calculated using the sparse multi-class cross-entropy loss function; The model is trained using the Adam optimizer, which minimizes the loss function using gradient descent and updates the model's weights.

4. The multi-stage training method for a text classification model based on multiple views as described in claim 1, characterized in that, During the second stage of intermediate training, the cross-entropy loss function is selected to calculate the difference between the model's predicted class and the pseudo-label. The Adam optimizer is used for backpropagation to calculate the gradient and update the model parameters.

5. The multi-stage training method for a text classification model based on multiple views as described in claim 1, characterized in that, Based on the correlation between the text data used in the third stage of training, the trained model is associated, and the associated model is called for auxiliary verification when the trained model is used for classification.

Images