Noise data training method based on weak supervision
By using weakly supervised noise data training method on text data that are not fully annotated and contains noise, the prediction model is trained and pseudo-labels are generated, and the pseudo-labels are corrected using external knowledge base checksums. Finally, the model weight is updated through iterative training, and the problem of limited performance of text mining models in the existing technology is solved, achieving efficient and accurate information extraction.
Patent Information
- Application Number
- CN202411462992.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-03-25
AI Technical Summary
When the prior art processes text data with incomplete annotation and noise, it is difficult to effectively train the text mining model, resulting in limited model performance and unable to meet the efficiency and effect requirements of large-scale data processing.
A weak supervision-based noise data training method is adopted, and the prediction model is trained and pseudo-labels are generated by optimizing the loss function and dynamic weighting mechanism. The pseudo-labels are corrected using external knowledge base checksums, and finally the model weight is updated through iterative training to improve the stability and generalization performance of the model.
While reducing dependence on high-quality annotated data, it improves the accuracy of information extraction of text mining models, enhances the stability and generalization capabilities of the model, and adapts to the needs of large-scale text data processing.
Smart Images
Figure CN119691442A_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a weakly supervised noise data training method, in particular a method for training a text mining model on incompletely labeled and noisy data, belonging to the technical field of text mining and data processing. Background Art
[0002] In the field of text mining, processing and analyzing large-scale text data is a common requirement, especially in building knowledge graphs, automatic question-answering systems, and other natural language processing applications. These tasks usually require extracting accurate and meaningful information from text, such as entity recognition, relationship extraction, and sentiment analysis. However, text data is usually unstructured and contains a lot of noise and incomplete annotations, which poses a great challenge to the accuracy and efficiency of data processing.
[0003] Ideally, text mining and information extraction techniques rely on high-quality annotated data. This data is manually annotated by experts to provide a benchmark for model training and validation. However, manual annotation is a time-intensive and costly process, and it is extremely difficult to achieve high coverage and consistency in large-scale datasets. Subjectivity and inconsistency in the annotation process often lead to data noise, which affects the performance of the trained model.
[0004] In order to improve the efficiency of annotation, some studies have adopted semi-automatic annotation methods, such as rule-based automatic annotation or machine learning-assisted annotation systems. These systems automatically generate labels through pre-defined rules or trained models, which are then verified and corrected by human annotators. Although this method can reduce the manual burden to a certain extent, there is still the problem of inaccurate generated labels or missing important information, especially when dealing with complex or ambiguous texts.
[0005] Given the difficulty in obtaining large amounts of high-quality labeled data, researchers have begun to explore the use of weakly supervised and unsupervised learning methods to train text mining models. These methods do not rely on large amounts of labeled data, but instead use algorithms to find patterns and relationships in large amounts of unlabeled data. For example, weakly supervised learning may use a small amount of labeled data or heuristic rules as the starting point for training, while unsupervised learning relies entirely on the structural characteristics of the data itself.
[0006] As the amount of data continues to increase and processing requirements increase, traditional text mining methods are gradually unable to meet the dual requirements of efficiency and effectiveness. This has driven the research of new methods, especially those that can perform well on incomplete or noisy annotated data. These methods need to be able to adapt to the incompleteness of data, resist the interference of noise, and provide explainable model predictions so that users can understand and trust the decision-making process of the model.
[0007] To this end, a noisy data training method based on weak supervision is designed to overcome the above problems. Summary of the invention
[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a noise data training method based on weak supervision to improve the feasibility and effectiveness of text mining in practical applications. This method utilizes the existing small amount of labeled data and machine learning algorithms to optimize the process of learning from noise data and reduce the dependence on a large amount of high-quality labeled data, thereby improving the accuracy of information extraction while maintaining efficient data processing.
[0009] The present invention is implemented by the following technical solution: a noise data training method based on weak supervision, the method comprising the following steps:
[0010] 1) Training prediction model based on noisy data:
[0011] The preliminary prediction model is trained using an incompletely labeled noisy training set. The training process uses supervised learning. Although there is label noise in the training data, the model can learn useful features from it and capture the positive information in the data through the optimized loss function. In order to ensure that the model can effectively adapt to these noises, a special loss function weighting mechanism is used to give smaller weights to samples with higher uncertainty and higher weights to samples with high confidence, so that the model pays more attention to effective information during the training process and reduces the interference of noise data. The model trained in this step is the basic prediction model, called model M;
[0012] 2) Generate pseudo labels using the prediction model:
[0013] After obtaining the initially trained prediction model M, it is applied to unlabeled or incompletely labeled data to generate pseudo labels. All unlabeled entity pairs in the training set are predicted by model M to obtain the predicted labels of these entity pairs, i.e. pseudo labels. These pseudo labels are equivalent to the inference of the unlabeled data by model M, which can expand the labeling coverage of the data, thereby forming richer training data.
[0014] For each pseudo-label, the model will give a confidence score, indicating the reliability of its prediction of the label. Only when the confidence score exceeds the preset threshold will the pseudo-label be retained for subsequent training. This can effectively reduce the number of erroneous pseudo-labels and ensure that the generated pseudo-labels are as accurate as possible.
[0015] 3) Use external knowledge base to verify and correct pseudo labels based on conflict identification
[0016] Since the pseudo-label generation process depends on the output of the prediction model, there may be errors in labeling. Therefore, an external knowledge base is introduced to correct the generated pseudo-labels to ensure that they conform to reasonable logical relationships. If two entity pairs in the pseudo-labels conflict, the conflict is identified through the external knowledge base and corresponding adjustments are made. The construction of the external knowledge base is based on domain knowledge and statistical laws in the data set. These rules include constraints such as transitivity, symmetry, and exclusivity of entity relationships. During the verification and correction process, for each conflict, the system will find the minimum number of pseudo-labels for correction, so that the corrected label set is consistent with the external knowledge base. This method can effectively reduce unnecessary modifications and keep the labels as correct as possible. Different costs are assigned to different pseudo-label corrections, and the correction scheme with lower costs is preferred to maintain the quality of the pseudo-labels to the greatest extent.
[0017] 4) Use pseudo labels to train the weight update model
[0018] After the pseudo labels are verified and corrected, they are used to retrain the model. By using the corrected pseudo labels for iterative training, the model weights are continuously updated to better fit the real relationship in the data. After each iteration, the model can better capture the subtle features in the data, thereby gradually improving the overall prediction performance. In order to prevent the model from falling into the local optimum during the iterative training process, after each round of training, the performance of the model is monitored on the validation set. If the performance of the validation set does not improve after multiple iterations, the training is stopped to avoid overfitting. In addition, the learning rate decay strategy is used to gradually reduce the learning rate as the training progresses, so that the model can adjust the parameters more finely when it is close to the optimal solution.
[0019] 5) Use the weights of the weight update model to update the weights of the prediction model
[0020] In order to further improve the stability and generalization performance of the model, a dynamic weight update strategy is adopted. Specifically, in each iteration of model M, the exponential moving average (EMA) is used to smoothly update the model weight to reduce fluctuations during the training process. This method can ensure that the model maintains strong stability while being continuously updated, and avoid a sudden drop in model performance caused by some unstable labels. In addition, a model fusion method is also used to fuse different versions of models obtained from multiple iterations. The prediction results of multiple models are combined by weighted average or voting to obtain more stable and accurate predictions. This model fusion strategy can effectively reduce the deviation of a single model and improve the performance of the overall system.
[0021] 6) Model evaluation and optimization
[0022] After each round of training, the model is comprehensively evaluated. Evaluation indicators include precision, recall rate, and F1 value. The model is further optimized based on the evaluation results, and the model parameters and training strategies are adjusted to ensure that the final performance of the model is optimal. The model is evaluated not only on the training set, but also on the development set to ensure that the model has good generalization capabilities. In order to better understand the behavior of the model, this module helps researchers understand the features and rules that the model relies on when making decisions by analyzing the prediction basis of the model, thereby further optimizing the model structure and training process. Visualization tools can be used to display the activation of the model on different input samples, as well as the degree of dependence of the model on specific features, so as to discover potential problems and make targeted improvements.
[0023] Preferably, the formula of the loss function in step 1) is as follows:
[0024]
[0025] Among them, w i is the weight of sample i, which is dynamically adjusted according to the confidence of the sample, l i is the loss of sample i. In order to further enhance the generalization ability of the model, data enhancement technology is also used to process noisy data, such as synonym replacement, random cropping and expansion of text. These data enhancement methods help the model better understand the diversity of text and enhance its robustness to different forms of expression, making it more stable when facing unknown data.
[0026] Preferably, the screening conditions for pseudo labels in step 2) are:
[0027] y=p(y i |x i )>τ,
[0028] Among them, y i is the model score of sample i, x i is the sample input, τ is the pseudo-label binarization threshold. In addition, the present invention also adopts an ensemble learning-based method to jointly generate pseudo-labels using models initialized with multiple different parameters to reduce the impact of a single model bias on the quality of pseudo-labels.
[0029] Preferably, the fusion prediction formula in step 5) is:
[0030]
[0031] Among them, K is the number of models involved in the fusion, is the prediction result of the kth model.
[0032] The beneficial effects of the present invention are as follows:
[0033] The present invention designs a weakly supervised noise data training method to improve the feasibility and effectiveness of text mining in practical applications. This method utilizes a small amount of existing labeled data and machine learning algorithms to optimize the process of learning from noise data and reduce dependence on a large amount of high-quality labeled data, thereby improving the accuracy of information extraction while maintaining efficient data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a schematic diagram of the overall process of the present invention.
[0035] Figure 2 This figure shows how the pseudo labels generated by the prediction model are verified and corrected using an external knowledge base in the present invention. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0037] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0038] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1-2 As shown, a noise data training method based on weak supervision comprises the following steps:
[0039] 1) Training prediction model based on noisy data:
[0040] The preliminary prediction model is trained using an incompletely labeled noisy training set. The training process uses supervised learning. Although there is label noise in the training data, the model can learn useful features from it and capture the positive information in the data through the optimized loss function. In order to ensure that the model can effectively adapt to these noises, a special loss function weighting mechanism is used to give smaller weights to samples with higher uncertainty and higher weights to samples with high confidence, so that the model pays more attention to effective information during the training process and reduces the interference of noise data. The model trained in this step is the basic prediction model, called model M;
[0041] 2) Generate pseudo labels using the prediction model:
[0042] After obtaining the initially trained prediction model M, it is applied to unlabeled or incompletely labeled data to generate pseudo labels. All unlabeled entity pairs in the training set are predicted by model M to obtain the predicted labels of these entity pairs, i.e. pseudo labels. These pseudo labels are equivalent to the inference of the unlabeled data by model M, which can expand the labeling coverage of the data, thereby forming richer training data.
[0043] For each pseudo-label, the model will give a confidence score, indicating the reliability of its prediction of the label. Only when the confidence score exceeds the preset threshold will the pseudo-label be retained for subsequent training. This can effectively reduce the number of erroneous pseudo-labels and ensure that the generated pseudo-labels are as accurate as possible.
[0044] 3) Use external knowledge base to verify and correct pseudo labels based on conflict identification
[0045] Since the pseudo-label generation process depends on the output of the prediction model, there may be errors in labeling. Therefore, an external knowledge base is introduced to correct the generated pseudo-labels to ensure that they conform to reasonable logical relationships. If two entity pairs in the pseudo-labels conflict, the conflict is identified through the external knowledge base and corresponding adjustments are made. The construction of the external knowledge base is based on domain knowledge and statistical laws in the data set. These rules include constraints such as transitivity, symmetry, and exclusivity of entity relationships. During the verification and correction process, for each conflict, the system will find the minimum number of pseudo-labels for correction, so that the corrected label set is consistent with the external knowledge base. This method can effectively reduce unnecessary modifications and keep the labels as correct as possible. Different costs are assigned to different pseudo-label corrections, and the correction scheme with lower costs is preferred to maintain the quality of the pseudo-labels to the greatest extent.
[0046] 4) Use pseudo labels to train the weight update model
[0047] After the pseudo labels are verified and corrected, they are used to retrain the model. By using the corrected pseudo labels for iterative training, the model weights are continuously updated to better fit the real relationship in the data. After each iteration, the model can better capture the subtle features in the data, thereby gradually improving the overall prediction performance. In order to prevent the model from falling into the local optimum during the iterative training process, after each round of training, the performance of the model is monitored on the validation set. If the performance of the validation set does not improve after multiple iterations, the training is stopped to avoid overfitting. In addition, the learning rate decay strategy is used to gradually reduce the learning rate as the training progresses, so that the model can adjust the parameters more finely when it is close to the optimal solution.
[0048] 5) Use the weights of the weight update model to update the weights of the prediction model
[0049] In order to further improve the stability and generalization performance of the model, a dynamic weight update strategy is adopted. Specifically, in each iteration of model M, the exponential moving average (EMA) is used to smoothly update the model weight to reduce fluctuations during the training process. This method can ensure that the model maintains strong stability while being continuously updated, and avoid a sudden drop in model performance caused by some unstable labels. In addition, a model fusion method is also used to fuse different versions of models obtained from multiple iterations. The prediction results of multiple models are combined by weighted average or voting to obtain more stable and accurate predictions. This model fusion strategy can effectively reduce the deviation of a single model and improve the performance of the overall system.
[0050] 6) Model evaluation and optimization
[0051] After each round of training, the model is comprehensively evaluated. Evaluation indicators include precision, recall rate, and F1 value. The model is further optimized based on the evaluation results, and the model parameters and training strategies are adjusted to ensure that the final performance of the model is optimal. The model is evaluated not only on the training set, but also on the development set to ensure that the model has good generalization capabilities. In order to better understand the behavior of the model, this module helps researchers understand the features and rules that the model relies on when making decisions by analyzing the prediction basis of the model, thereby further optimizing the model structure and training process. Visualization tools can be used to display the activation of the model on different input samples, as well as the degree of dependence of the model on specific features, so as to discover potential problems and make targeted improvements.
[0052] The formula of the loss function in step 1) is as follows:
[0053]
[0054] Among them, w i is the weight of sample i, which is dynamically adjusted according to the confidence of the sample, l i is the loss of sample i. In order to further enhance the generalization ability of the model, data enhancement technology is also used to process noisy data, such as synonym replacement, random cropping and expansion of text. These data enhancement methods help the model better understand the diversity of text and enhance its robustness to different forms of expression, making it more stable when facing unknown data.
[0055] The screening conditions for pseudo labels in step 2) are:
[0056] y=p((y i |x i )>τ,
[0057] Among them, yi is the model score of sample i, x i is the sample input, τ is the pseudo-label binarization threshold. In addition, the present invention also adopts an ensemble learning-based method to jointly generate pseudo-labels using models initialized with multiple different parameters to reduce the impact of a single model bias on the quality of pseudo-labels.
[0058] The fusion prediction formula in step 5) is:
[0059]
[0060] Among them, K is the number of models involved in the fusion, is the prediction result of the kth model.
[0061] The present invention uses weakly supervised learning to effectively extract useful information from noisy text data, which is crucial for building knowledge graphs and enhancing the application of question-answering systems. The present invention can improve the feasibility and effectiveness of text mining in practical applications. This method uses a small amount of existing annotated data and machine learning algorithms to optimize the process of learning from noisy data and reduce the dependence on a large amount of high-quality annotated data, thereby maintaining efficient data processing while improving the accuracy of information extraction.
[0062] The specific embodiments described herein are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A noise data training method based on weak supervision, characterized in that: The method comprises the following steps: 1) Training prediction model based on noisy data: The preliminary prediction model is trained using an incompletely labeled noisy training set. The training process uses supervised learning. Although there is label noise in the training data, the model can learn useful features from it and capture the positive information in the data through the optimized loss function. In order to ensure that the model can effectively adapt to these noises, a special loss function weighting mechanism is used to give smaller weights to samples with higher uncertainty and higher weights to samples with high confidence, so that the model pays more attention to effective information during the training process and reduces the interference of noise data. The model trained in this step is the basic prediction model, called model M; 2) Generate pseudo labels using the prediction model: After obtaining the initially trained prediction model M, it is applied to unlabeled or incompletely labeled data to generate pseudo labels. All unlabeled entity pairs in the training set are predicted by model M to obtain the predicted labels of these entity pairs, i.e. pseudo labels. These pseudo labels are equivalent to the inference of the unlabeled data by model M, which can expand the labeling coverage of the data, thereby forming richer training data. For each pseudo-label, the model will give a confidence score, indicating the reliability of its prediction of the label. Only when the confidence score exceeds the preset threshold will the pseudo-label be retained for subsequent training. This can effectively reduce the number of erroneous pseudo-labels and ensure that the generated pseudo-labels are as accurate as possible. 3) Use external knowledge base to verify and correct pseudo labels based on conflict identification Since the pseudo-label generation process depends on the output of the prediction model, there may be errors in labeling. Therefore, an external knowledge base is introduced to correct the generated pseudo-labels to ensure that they conform to reasonable logical relationships. If two entity pairs in the pseudo-labels conflict, the conflict is identified through the external knowledge base and corresponding adjustments are made. The construction of the external knowledge base is based on domain knowledge and statistical laws in the data set. These rules include transitivity, symmetry, and exclusivity constraints on entity relationships. During the verification and correction process, for each conflict, the system will find the minimum number of pseudo-labels for correction, so that the corrected label set is consistent with the external knowledge base. This method can effectively reduce unnecessary modifications and keep the labels as correct as possible. Different costs are assigned to different pseudo-label corrections, and the correction scheme with lower costs is preferred to maintain the quality of the pseudo-labels to the greatest extent. 4) Use pseudo labels to train the weight update model After the pseudo labels are verified and corrected, they are used to retrain the model. By using the corrected pseudo labels for iterative training, the model weights are continuously updated to better fit the real relationship in the data. After each iteration, the model can better capture the subtle features in the data, thereby gradually improving the overall prediction performance. In order to prevent the model from falling into the local optimum during the iterative training process, after each round of training, the performance of the model is monitored on the validation set. If the performance of the validation set does not improve after multiple iterations, the training is stopped to avoid overfitting. In addition, the learning rate decay strategy is used to gradually reduce the learning rate as the training progresses, so that the model can adjust the parameters more finely when it is close to the optimal solution. 5) Use the weights of the weight update model to update the weights of the prediction model In order to further improve the stability and generalization performance of the model, a dynamic weight update strategy is adopted. Specifically, in each iteration of model M, the exponential moving average (EMA) is used to smoothly update the model weight to reduce fluctuations during the training process. This method can ensure that the model maintains strong stability while being continuously updated, and avoid a sudden drop in model performance caused by some unstable labels. In addition, a model fusion method is also used to fuse different versions of models obtained from multiple iterations. The prediction results of multiple models are combined by weighted average or voting to obtain more stable and accurate predictions. This model fusion strategy can effectively reduce the deviation of a single model and improve the performance of the overall system. 6) Model evaluation and optimization After each round of training, the model is comprehensively evaluated. The evaluation indicators include precision, recall rate and F1 value. The model is further optimized through the evaluation results, and the model parameters and training strategies are adjusted to ensure that the final performance of the model is optimal. The model is evaluated not only for the training set, but also for the verification of the development set to ensure that the model has good generalization ability. In order to better understand the behavior of the model, this module helps researchers understand the features and rules that the model relies on when making decisions by analyzing the prediction basis of the model, so as to further optimize the model structure and training process. The model activation status on different input samples and the degree of model dependence on specific features can be displayed through visualization tools, so as to discover potential problems and make targeted improvements.
2. The noise data training method based on weak supervision according to claim 1, characterized in that: The formula of the loss function in step 1) is as follows: Among them, w i is the weight of sample i, which is dynamically adjusted according to the confidence of the sample, l i is the loss of sample i. In order to further enhance the generalization ability of the model, data enhancement technology is also used to process noisy data, such as synonym replacement, random cropping and expansion of text. These data enhancement methods help the model better understand the diversity of text and enhance its robustness to different forms of expression, making it more stable when facing unknown data.
3. The noise data training method based on weak supervision according to claim 1, characterized in that: The screening conditions for pseudo labels in step 2) are: y=p(y i |x i )>τ, Among them, y i is the model score of sample i, x i is the sample input, τ is the pseudo-label binarization threshold. In addition, the present invention also adopts an ensemble learning-based method to jointly generate pseudo-labels using models initialized with multiple different parameters to reduce the impact of a single model bias on the quality of pseudo-labels.
4. The noise data training method based on weak supervision according to claim 1, characterized in that: The fusion prediction formula in step 5) is: Among them, K is the number of models involved in the fusion, is the prediction result of the kth model.