Magic box correlation training sample enhancement and expansion method based on small amount of annotation data
By using a small amount of magic box correlation training sample enhancement and expansion method in search engines, combining automation and manual annotation, dynamic threshold setting and hybrid strategies, the challenge of magic box correlation optimization is solved and the model is improved abilities and accuracy to adapt to complex task scenarios.
Patent Information
- Application Number
- CN202410997735.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
In search engines, the optimization of magic box correlation faces challenges, especially in the automotive field. Traditional annotation methods take time, costly and require a large number of unbiased high-quality annotation samples. Training of deep semantic matching models is difficult to meet the generalization ability and accuracy requirements.
Using Magic Box Relevance Training Sample Enhancement and Expansion Method Based on a Small Number of Annotated Data, Dynamic Threshold Settings by Fast Construction of Training Framework, Manual Intervention and Sample Quality Improvement, Continuous Optimization and Sample Expansion, and Comparative Learning Sample Enhancement, Combined with Automation and Manual Annotation, Dynamic Threshold Settings Determine and mix strategies to generate a diverse training set.
The rapid generation of high-quality labeled data has been achieved, which improves the generalization ability and classification accuracy of the model. The accuracy of the negative case judgment of the correlation model has been increased by 15.72%, and the negative case recall rate has been increased by 37.23%, reducing the risk of overfitting and adapting Complex and varied task scenarios.
Smart Images

Figure CN118940089A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of search engines, and in particular is a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data. Background Art
[0002] In search engines and recommendation systems, search relevance is one of the important indicators for measuring system performance. It is directly related to whether users can find the information they need quickly and accurately. Especially in search scenarios in the automotive field, such as Autohome and other applications, the importance of search relevance is more prominent. Among them, the magic box, as a special form of search result display, is at the top of the search results and carries the main search needs. The quality of its relevance directly determines the user's search experience and satisfaction. However, the optimization of the magic box relevance faces many challenges. First of all, the search terms in the car APP are often short and concise, but they may contain rich information, such as car model, brand, displacement, etc. Therefore, when judging the relevance of the magic box and the search terms, in addition to considering the match between the search terms and the magic box title, it is also necessary to comprehensively consider the search terms and the magic box type, the car model title displayed in the magic box, the car This complex matching logic significantly increases the difficulty of relevance annotation. Secondly, since the four major categories of magic boxes (car model, car series, brand, manufacturer) only show the only correct result under a single search term, this setting has almost zero tolerance for relevance errors. Once the model predicts an error, it will directly lead to the user being unable to find the required information, seriously affecting the user experience. In traditional annotation methods, although manual annotation can ensure the high quality of samples, its time-consuming and costly shortcomings cannot be ignored. With the widespread application of deep learning technology in the field of natural language processing, search relevance optimization based on deep semantic matching models has become a new solution. However, such models require a large number of unbiased high-quality annotated samples for training to ensure their generalization ability and accuracy. Therefore, they need to be improved and optimized. Summary of the invention
[0003] The purpose of the present invention is to provide a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data to solve the problems raised in the above background technology.
[0004] In order to achieve the above-mentioned object, the present invention provides the following technical solution: a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data, the specific steps of the method include:
[0005] Step 1, early stage of the project: quickly build a training framework and generate basic samples;
[0006] Step 2, mid-term of the project: manual intervention and sample quality improvement;
[0007] Step 3, project improvement phase: continuous optimization and sample expansion;
[0008] Step 4: Model-level optimization: comparative learning sample enhancement.
[0009] Preferably, the specific steps of quickly constructing a training framework and generating basic samples in step 1 include:
[0010] A1, establish a fast annotation strategy: based on the observation that the click rate of the search terms under the magic box is basically positively correlated with the relevance, the click rate is used as the preliminary indicator of relevance annotation;
[0011] A2, Advantages of posterior click rate annotation:
[0012] A2-1, Efficiency: Quickly generate a large number of training samples and accelerate the model training cycle;
[0013] A2-2, unbiased: reflects the real behavior of users and reduces human bias;
[0014] A3, Dealing with shortcomings:
[0015] Data noise: Be aware that data quality is not high and leave room for subsequent optimization.
[0016] Preferably, the specific steps of establishing a rapid labeling strategy in A1 include:
[0017] B1, using click-through rate data as a preliminary indicator:
[0018] B1-1, Observation basis: Based on historical data analysis, it is found that the click rate of Magic Box under specific search terms is basically positively correlated with the relevance of its content;
[0019] B1-2, annotation method: click-through rate is used as the preliminary indicator of relevance annotation;
[0020] B2, data cleaning to reduce noise:
[0021] B2-1, low-frequency exposure processing: remove samples with low-frequency exposure to reduce the noise introduced by unequal display opportunities;
[0022] B2-2, medium click rate sample screening: remove these samples with medium click rates to simplify the annotation process and improve annotation efficiency;
[0023] B3, set dynamic thresholds to adapt to data differences:
[0024] B3-1, Difference considerations: Pay attention to the significant differences in click-through rates between different search terms and magic boxes;
[0025] B3-2, Dynamic Threshold Setting: Dynamically set the click rate entry threshold for different search terms and magic boxes to more accurately reflect the differences;
[0026] B4, expand the content to improve the comprehensiveness of the annotation:
[0027] B4-1, multi-dimensional consideration: In addition to the main indicator of click-through rate, other dimensions of information are introduced to assist the annotation process;
[0028] B4-2, Comprehensive Annotation: By comprehensively considering information from multiple dimensions, the correlation between the magic box and the search terms can be evaluated more comprehensively and accurately, and a more representative training sample set can be generated.
[0029] Preferably, the specific steps of manual intervention and sample quality improvement in step 2 include:
[0030] C1, model scoring and screening of difficult samples:
[0031] C1-1, model initial screening: use the click-through rate annotation-based model to score the log data;
[0032] C1-2, difficult sample identification: filter out samples with ambiguous scores or inconsistent click-through rates as the focus of manual labeling;
[0033] C2, manual labeling and sample space solution: Manually label the difficult-to-separate samples, add easy-to-separate samples, use a large number of click-through rate labeled samples to pre-train the model, and then fine-tune it based on the difficult-to-separate and easy-to-separate samples.
[0034] Preferably, the specific steps of manual labeling and sample space solution in C2 include:
[0035] D1, high-quality annotation: manual annotation of difficult samples:
[0036] D1-1, Identify difficult samples: Preliminarily screen out a batch of samples that are difficult to automatically classify through the algorithm, and analyze samples that are located at the boundary of categories and have fuzzy features;
[0037] D1-2, manual review and labeling: Submit difficult samples to a manual labeling team with professional knowledge and experience for review and labeling;
[0038] D1-3, feedback loop: Feedback the results of manual annotation back to the model to adjust and optimize the automatic annotation algorithm, forming a virtuous cycle;
[0039] D2, sample diversity enhancement: adding easy-to-separate samples:
[0040] D2-1, extreme case addition: In order to increase the diversity of samples, some extreme cases are added, including samples with extremely high and low click rates;
[0041] D2-2, query and magic box name matching samples: In addition to extreme cases, add some samples where the query and magic box name directly match and are highly similar;
[0042] D2-3, mixed strategy: easy-to-classify samples and difficult-to-classify samples are mixed in a certain proportion to form a diversified training set;
[0043] D3, pre-training and fine-tuning:
[0044] D3-1, Pre-training with massive click-through rate labeled samples: Pre-train the model using massive samples automatically labeled based on click-through rate;
[0045] D3-2, fine-tuning with hard-to-separate and easy-to-separate samples: Based on pre-training, fine-tune the model using manually annotated high-quality hard-to-separate samples and enhanced easy-to-separate samples;
[0046] D3-3, iterative optimization: Continuously optimize the performance of the model through multiple iterative pre-training and fine-tuning processes.
[0047] Preferably, the specific steps of continuous optimization and sample expansion in step 3 include:
[0048] E1, targeted artificial sample construction:
[0049] E1-1, problem-oriented: Construct specific samples for specific problems encountered in model optimization, including magic box aliasing, over-recall, and multi-magic box mapping;
[0050] E1-2, maintain distribution: accurately introduce problem-related samples without affecting the overall sample distribution;
[0051] E2, automated sample screening based on confidence learning:
[0052] E2-1, iterative optimization: Use high-quality samples as seeds, automatically screen TN / TP samples in large-scale CTR samples, and iteratively optimize the training set;
[0053] E2-2, improve efficiency: reduce manual dependence and speed up the sample expansion process;
[0054] E3, strong rule sample correction:
[0055] E3-1, rule-driven: Use business knowledge and strong rules to modify training samples and further improve sample quality;
[0056] E4, sample mixing and finite element analysis: Mix samples that are difficult to classify, easy to classify, manually constructed, automatically labeled, and strongly rule-labeled in a certain proportion and use them in the finite element analysis model.
[0057]
[0058] Preferably, the specific steps of sample mixing and filtering in E4 include:
[0059] F1, hybrid strategy: The core is to combine samples from different sources and characteristics in a certain proportion to form a diverse training set for fine-tuning the model, including:
[0060] F1-1, difficult samples: samples that are located at the boundary of the category or have fuzzy features and are difficult to be accurately classified by the model;
[0061] F1-2, easy-to-classify samples: samples with clear features that are easy to be classified by the model;
[0062] F1-3, artificially constructed samples: artificially constructed samples are created based on domain knowledge and expert opinions;
[0063] F1-4, automated annotation samples: samples generated using automated annotation tools or algorithms;
[0064] F1-5, strong rule-annotated samples: samples annotated based on strict business rules and logic;
[0065] F2, balance quality and quantity:
[0066] F2-1, ensuring representativeness: training samples should be able to comprehensively cover all possible situations of the target task, including common and rare scenarios;
[0067] F2-2, noise control: Automatically labeled samples contain certain noise or incorrect labels. When mixing these samples, corresponding measures need to be taken to reduce the impact of noise;
[0068] F2-3, principle of appropriate amount: when mixing samples, the quantity range needs to be determined according to the actual situation;
[0069] F2-4, dynamic adjustment: During the training process, the mixing ratio and number of samples are dynamically adjusted according to the performance of the model on the validation set.
[0070] Preferably, the specific steps of enhancing the comparative learning samples in step 4 include:
[0071] G1, contrastive learning application:
[0072] G1-1, SimCSE framework: The unsupervised SimCSE model is used for sample enhancement. The embedding generated by Dropout2 times of the same sample is used as a positive sample and combined with other samples in the batch as a negative sample pair;
[0073] G1-2, loss function optimization: design a loss function to maximize the similarity of positive sample pairs embedding and minimize the similarity of negative sample pairs;
[0074] G2, improves generalization ability:
[0075] Enhance sample diversity: By comparing the samples generated by learning, we can increase the diversity and complexity of the training set and improve the model's ability to generalize to unseen data.
[0076] The beneficial effects of the present invention are as follows:
[0077] 1. The present invention realizes the rapid processing and high-quality labeling of large amounts of data by combining automatic labeling and manual labeling. The automatic labeling tool can quickly process massive amounts of data and reduce the burden of manual labeling, while manual labeling can ensure the accuracy of difficult-to-distinguish samples and key samples. At the same time, by setting dynamic thresholds and hybrid strategies, the diversity and representativeness of the labeled data are further improved, so that the model can learn more comprehensive and in-depth features. This efficient data utilization and labeling strategy provides a solid data foundation for model training. Through sample construction and enhancement schemes, the cumulative negative example judgment accuracy of the correlation model is improved by 15.72pp, and the negative example recall rate is improved by 37.23pp.
[0078] 2. The present invention constructs a diversified training set by mixing multiple types of samples such as difficult-to-separate, easy-to-separate, manually constructed, automatically labeled, and strongly rule-labeled. This diversified training set not only helps the model learn a broader and deeper feature representation, but also can reduce the risk of overfitting to a certain extent and improve the generalization ability of the model. At the same time, the detailed labeling and manual review of difficult-to-separate samples further improve the classification accuracy of the model near the category boundaries. Therefore, this method can train a model with both strong generalization ability and high accuracy.
[0079] 3. The present invention introduces a flexible adjustment and optimization mechanism in the training process. By dynamically adjusting the sample mixing ratio and quantity, and fine-tuning the model according to its performance on the validation set, the training process can be optimized in real time to ensure that the model always remains in the best state during the training process. In addition, the method also allows the labeling strategy and sample selection strategy to be adjusted according to actual needs to adapt to different tasks and data sets. This flexible adjustment and optimization mechanism enables the method to cope with various complex and changeable task scenarios, ensuring that the performance and effect of the model always remain at a high level. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] Figure 1 This is a flow chart of the training sample enhancement and expansion method of the present invention;
[0081] Figure 2 It is the sample enhancement logic of the SimCSE model of the present invention;
[0082] Figure 3 Flow chart of the output scheme of correlation training samples in different stages of the present invention
[0083] Figure 4 This is a sample construction example diagram of the present invention. DETAILED DESCRIPTION
[0084] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0085] like Figures 1 to 4 As shown, the embodiment of the present invention provides a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data, and the specific steps of the method include:
[0086] Step 1, early stage of the project: quickly build a training framework and generate basic samples;
[0087] Step 2, mid-term of the project: manual intervention and sample quality improvement;
[0088] Step 3, project improvement phase: continuous optimization and sample expansion;
[0089] Step 4: Model-level optimization: comparative learning sample enhancement.
[0090] The specific steps of quickly building a training framework and generating basic samples in step 1 include:
[0091] A1, establish a fast annotation strategy: based on the observation that the click rate of the search terms under the magic box is basically positively correlated with the relevance, the click rate is used as the preliminary indicator of relevance annotation;
[0092] A2, Advantages of posterior click rate annotation:
[0093] A2-1, Efficiency: Quickly generate a large number of training samples and accelerate the model training cycle;
[0094] A2-2, unbiased: reflects the real behavior of users and reduces human bias;
[0095] A3, Dealing with shortcomings:
[0096] Data noise: Be aware that data quality is not high and leave room for subsequent optimization.
[0097] Among them, the specific steps of establishing a rapid annotation strategy in A1 include:
[0098] B1, using click-through rate data as a preliminary indicator:
[0099] B1-1, Observation basis: Based on historical data analysis, it is found that the click rate of Magic Box under specific search terms is basically positively correlated with the relevance of its content. The results provide a fast and relatively reliable basis for annotation;
[0100] B1-2, annotation method: Click-through rate is used as the preliminary indicator of relevance annotation. Specifically, the magic boxes with higher click-through rates are regarded as samples that are more relevant to the search terms, while those with lower click-through rates are regarded as irrelevant or weakly relevant samples;
[0101] B2, data cleaning to reduce noise:
[0102] B2-1, low-frequency exposure processing: remove samples with low frequency exposure, reduce the noise introduced by unequal display opportunities, reduce the low click-through rate of the magic box due to limited display opportunities, and avoid the samples not being able to truly reflect the correlation between the magic box and the search terms;
[0103] B2-2, medium click-through rate sample screening: remove these samples with medium click-through rate, simplify the annotation process and improve the annotation efficiency, and reduce the samples with medium click-through rate that contain more uncertainty;
[0104] B3, set dynamic thresholds to adapt to data differences:
[0105] B3-1, Difference considerations: Pay attention to the significant differences in click-through rates between different search terms and magic boxes. For example, the click-through rates of magic boxes under certain popular search terms are generally high, while the click-through rates of magic boxes under unpopular search terms may be low due to lack of user interest;
[0106] B3-2, Dynamic Threshold Setting: Dynamically set the click rate entry threshold for different search terms and magic boxes to more accurately reflect the differences. By counting the click rate distribution of magic boxes under each search term, we can set a suitable threshold for each search term or magic box combination to distinguish relevant and irrelevant samples.
[0107] B4, expand the content to improve the comprehensiveness of the annotation:
[0108] B4-1, multi-dimensional consideration: In addition to the main indicator of click-through rate, other dimensions of information are introduced to assist the annotation process. For example, user behavior data (such as dwell time, scroll depth, etc.) is analyzed to evaluate the user's interest in the content of the magic box; or the text similarity algorithm is used to compare the correlation between the search term and the magic box title and description;
[0109] B4-2, Comprehensive Annotation: By comprehensively considering information from multiple dimensions, the correlation between the magic box and the search terms can be evaluated more comprehensively and accurately, and a more representative training sample set can be generated, which not only helps to improve the generalization ability of the model, but also makes the model closer to the actual needs of users.
[0110] Among them, the specific steps of manual intervention and sample quality improvement in step 2 include:
[0111] C1, model scoring and screening of difficult samples:
[0112] C1-1, model initial screening: use the click-through rate annotation-based model to score the log data;
[0113] C1-2, difficult sample identification: filter out samples with ambiguous scores or inconsistent click-through rates as the focus of manual labeling;
[0114] C2, manual labeling and sample space solution: Manually label the difficult-to-separate samples, add easy-to-separate samples, use a large number of click-through rate labeled samples to pre-train the model, and then fine-tune it based on the difficult-to-separate and easy-to-separate samples.
[0115] Among them, the specific steps of manual labeling and sample space solution in C2 include:
[0116] D1, high-quality annotation: manual annotation of difficult samples:
[0117] D1-1, Identify difficult samples: Preliminarily screen out a batch of samples that are difficult to automatically classify through algorithms, and analyze samples that are located at the boundary of categories and have fuzzy features. For example, in the relevance labeling task, samples with a click rate close to the set threshold are often difficult to automatically determine whether they are truly relevant;
[0118] D1-2, manual review and labeling: Submit difficult samples to a manual labeling team with professional knowledge and experience for review and labeling. Manual labeling can ensure the accuracy and consistency of sample labels, thereby improving the overall sample quality;
[0119] D1-3, Feedback loop: Feedback the results of manual annotation back to the model to adjust and optimize the automatic annotation algorithm, forming a virtuous cycle; this can not only improve the accuracy of automatic annotation, but also reduce the reliance on manual annotation in the future;
[0120] D2, sample diversity enhancement: adding easy-to-separate samples:
[0121] D2-1, extreme case addition: In order to increase the diversity of samples, some extreme cases are added, including samples with extremely high and low click rates. Although these samples are easy to classify, they can help the model learn richer feature representations and decision boundaries;
[0122] D2-2, query and magic box name matching samples: In addition to extreme cases, add some samples where the query and magic box names directly match and are highly similar. Such samples usually have a high correlation, which helps the model learn the basic rules for correlation judgment;
[0123] D2-3, mixed strategy: easy-to-classify samples and difficult-to-classify samples are mixed in a certain proportion to form a diversified training set. This can not only ensure the generalization ability of the model, but also improve its performance in complex scenarios;
[0124] D3, pre-training and fine-tuning:
[0125] D3-1, Pre-training with massive click-through rate labeled samples: Pre-train the model using massive click-through rate automatically labeled samples; pre-training can help the model learn basic feature representation and preliminary classification capabilities, laying a good foundation for subsequent fine-tuning;
[0126] D3-2, fine-tuning with hard-to-separate and easy-to-separate samples: Based on pre-training, fine-tune the model using manually annotated high-quality hard-to-separate samples and enhanced easy-to-separate samples. During the fine-tuning process, the model will further optimize its classification boundaries and feature weights to adapt to a more complex and diverse sample space.
[0127] D3-3, iterative optimization: Through multiple iterative pre-training and fine-tuning processes, the performance of the model is continuously optimized, and the labeling strategy and sample selection strategy are adjusted in time according to the performance of the model on the validation set to ensure that the model can continuously improve its accuracy and generalization ability.
[0128] Among them, the specific steps of continuous optimization and sample expansion in step three include:
[0129] E1, targeted artificial sample construction:
[0130] E1-1, problem-oriented: Construct specific samples for specific problems encountered in model optimization, including magic box aliasing, over-recall, and multi-magic box mapping;
[0131] E1-2, maintain distribution: accurately introduce problem-related samples without affecting the overall sample distribution;
[0132] E2, automated sample screening based on confidence learning:
[0133] E2-1, iterative optimization: Use high-quality samples as seeds, automatically screen TN / TP samples in large-scale CTR samples, and iteratively optimize the training set;
[0134] E2-2, improve efficiency: reduce manual dependence and speed up the sample expansion process;
[0135] E3, strong rule sample correction:
[0136] E3-1, rule-driven: Use business knowledge and strong rules to modify training samples and further improve sample quality;
[0137] E4, sample mixing and finite element analysis: Mix samples that are difficult to classify, easy to classify, manually constructed, automatically labeled, and strongly rule-labeled in a certain proportion and use them in the finite element analysis model.
[0138]
[0139] Among them, the specific steps of sample mixing and fi netune in E4 include;
[0140] F1, hybrid strategy: The core is to combine samples from different sources and characteristics in a certain proportion to form a diverse training set for fine-tuning the model, including:
[0141] F1-1, difficult samples: samples that are located at the boundary of categories or have fuzzy features and are difficult to be accurately classified by the model. They are crucial to improving the model's ability to distinguish and identify boundaries.
[0142] F1-2, easy-to-classify samples: samples with clear features that are easy to be classified by the model, which helps the model learn basic classification rules and accelerate the training process;
[0143] F1-3, artificially constructed samples: artificially constructed samples are created based on domain knowledge and expert opinions to increase sample diversity or fill in gaps in certain specific scenarios;
[0144] F1-4, automated annotation samples: samples generated by automated annotation tools or algorithms to expand the size of the sample set;
[0145] F1-5, strongly rule-annotated samples: samples annotated based on strict business rules and logic. These samples usually have high accuracy but may be limited by the coverage of the rules;
[0146] F2, balance quality and quantity:
[0147] F2-1, ensuring representativeness: The training samples should be able to fully cover all possible situations of the target task, including common and rare scenarios, to ensure that the model can make reasonable predictions when faced with unknown inputs;
[0148] F2-2, noise control: Automatically labeled samples contain certain noise or incorrect labels. When mixing these samples, corresponding measures need to be taken to reduce the impact of noise, including setting thresholds, manual review, and using machine learning technology for cleaning;
[0149] F2-3, principle of appropriate amount: when mixing samples, the quantity range needs to be determined according to the actual situation;
[0150] F2-4, dynamic adjustment: During the training process, the mixing ratio and number of samples are dynamically adjusted according to the performance of the model on the validation set. If the model performs poorly on certain categories, the number of samples in that category can be increased or the mixing ratio can be adjusted to improve its accuracy.
[0151] Among them, the specific steps of contrast learning sample enhancement in step 4 include:
[0152] G1, contrastive learning application:
[0153] G1-1, SimCSE framework: The unsupervised SimCSE model is used for sample enhancement. The embedding generated by Dropout2 times of the same sample is used as a positive sample and combined with other samples in the batch as a negative sample pair;
[0154] G1-2, loss function optimization: design a loss function to maximize the similarity of positive sample pairs embedding and minimize the similarity of negative sample pairs;
[0155] G2, improves generalization ability:
[0156] Enhance sample diversity: By comparing the samples generated by learning, we can increase the diversity and complexity of the training set and improve the model's ability to generalize to unseen data.
[0157] The loss function used in the SimCSE model is:
[0158]
[0159] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0160] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data, characterized by: The specific steps of the method include: Step 1, early stage of the project: quickly build a training framework and generate basic samples; Step 2, mid-term of the project: manual intervention and sample quality improvement; Step 3, project improvement stage: continuous optimization and sample expansion; Step 4: Model-level optimization: comparative learning sample enhancement.
2. According to claim 1, a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data, characterized in that: The specific steps of quickly constructing the training framework and generating basic samples in step 1 include: A1, establish a fast annotation strategy: based on the observation that the click rate of the search terms under the magic box is basically positively correlated with the relevance, the click rate is used as the preliminary indicator of relevance annotation; A2, Advantages of posterior click rate annotation: A2-1, Efficiency: Quickly generate a large number of training samples and accelerate the model training cycle; A2-2, unbiased: reflects the real behavior of users and reduces human bias; A3, Dealing with shortcomings: Data noise: Be aware that data quality is not high and leave room for subsequent optimization.
3. According to claim 2, a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data, characterized in that: The specific steps of establishing the rapid annotation strategy in A1 include: B1, using click-through rate data as a preliminary indicator: B1-1, Observation basis: Based on historical data analysis, it is found that the click rate of Magic Box under specific search terms is basically positively correlated with the relevance of its content; B1-2, annotation method: click-through rate is used as the preliminary indicator of relevance annotation; B2, data cleaning to reduce noise: B2-1, low-frequency exposure processing: remove samples with low-frequency exposure to reduce the noise introduced by unequal display opportunities; B2-2, medium click rate sample screening: remove these samples with medium click rates to simplify the annotation process and improve annotation efficiency; B3, set dynamic thresholds to adapt to data differences: B3-1, Difference considerations: Pay attention to the significant differences in click-through rates between different search terms and magic boxes; B3-2, Dynamic Threshold Setting: Dynamically set the click rate entry threshold for different search terms and magic boxes to more accurately reflect the differences; B4, expand the content to improve the comprehensiveness of the annotation: B4-1, multi-dimensional consideration: In addition to the main indicator of click-through rate, other dimensions of information are introduced to assist the annotation process; B4-2, Comprehensive Annotation: By comprehensively considering information from multiple dimensions, the correlation between the magic box and the search terms can be evaluated more comprehensively and accurately, and a more representative training sample set can be generated.
4. According to claim 1, a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data, characterized in that: The specific steps of manual intervention and sample quality improvement in step 2 include: C1, model scoring and screening of difficult samples: C1-1, model initial screening: use the click-through rate annotation-based model to score the log data; C1-2, identification of difficult samples: filter out samples with ambiguous scores or inconsistent click-through rates as the focus of manual labeling; C2, manual labeling and sample space solution: Manually label the difficult-to-separate samples, add easy-to-separate samples, use a large number of click-through rate labeled samples to pre-train the model, and then fine-tune it based on the difficult-to-separate and easy-to-separate samples.
5. According to claim 4, a method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data, characterized in that: The specific steps of manual labeling and sample space solution in C2 include: D1, high-quality annotation: manual annotation of difficult samples: D1-1, Identify difficult samples: Preliminarily screen out a batch of samples that are difficult to automatically classify through the algorithm, and analyze samples that are located at the boundary of categories and have fuzzy features; D1-2, manual review and labeling: Submit difficult samples to a manual labeling team with professional knowledge and experience for review and labeling; D1-3, feedback loop: Feedback the results of manual annotation back to the model to adjust and optimize the automatic annotation algorithm, forming a virtuous cycle; D2, sample diversity enhancement: adding easy-to-separate samples: D2-1, extreme case addition: In order to increase the diversity of samples, some extreme cases are added, including samples with extremely high and low click rates; D2-2, query and magic box name matching samples: In addition to extreme cases, add some samples where the query and magic box name directly match and are highly similar; D2-3, mixed strategy: easy-to-classify samples and difficult-to-classify samples are mixed in a certain proportion to form a diversified training set; D3, pre-training and fine-tuning: D3-1, Pre-training with massive click-through rate labeled samples: Pre-train the model using massive samples automatically labeled based on click-through rate; D3-2, fine-tuning with hard-to-separate and easy-to-separate samples: Based on pre-training, fine-tune the model using manually annotated high-quality hard-to-separate samples and enhanced easy-to-separate samples; D3-3, iterative optimization: Continuously optimize the performance of the model through multiple iterative pre-training and fine-tuning processes.
6. The method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data according to claim 1, characterized in that: The specific steps of continuous optimization and sample expansion in step three include: E1, targeted artificial sample construction: E1-1, problem-oriented: Construct specific samples for specific problems encountered in model optimization, including magic box aliasing, over-recall, and multi-magic box mapping; E1-2, maintain distribution: accurately introduce problem-related samples without affecting the overall sample distribution; E2, automated sample screening based on confidence learning: E2-1, iterative optimization: Use high-quality samples as seeds, automatically screen TN / TP samples in large-scale CTR samples, and iteratively optimize the training set; E2-2, improve efficiency: reduce manual dependence and speed up the sample expansion process; E3, strong rule sample correction: E3-1, rule-driven: Use business knowledge and strong rules to modify training samples and further improve sample quality; E4, sample mixing and fine-tuning: Mix samples that are difficult to classify, easy to classify, manually constructed, automatically labeled, and strongly rule-labeled in a certain proportion for the fine-tuning model 7. The method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data according to claim 6, characterized in that: The specific steps of sample mixing and fine-tuning in E4 include: F1, hybrid strategy: The core is to combine samples from different sources and characteristics in a certain proportion to form a diverse training set for fine-tuning the model, including: F1-1, difficult samples: samples that are located at the boundary of the category or have fuzzy features and are difficult to be accurately classified by the model; F1-2, easy-to-classify samples: samples with clear features that are easy to be classified by the model; F1-3, artificially constructed samples: artificially constructed samples are created based on domain knowledge and expert opinions; F1-4, automated annotation samples: samples generated using automated annotation tools or algorithms; F1-5, strong rule-annotated samples: samples annotated based on strict business rules and logic; F2, balance quality and quantity: F2-1, ensuring representativeness: training samples should be able to comprehensively cover all possible situations of the target task, including common and rare scenarios; F2-2, noise control: Automatically labeled samples contain certain noise or incorrect labels. When mixing these samples, corresponding measures need to be taken to reduce the impact of noise; F2-3, principle of appropriate amount: when mixing samples, the quantity range needs to be determined according to the actual situation; F2-4, dynamic adjustment: During the training process, the mixing ratio and number of samples are dynamically adjusted according to the performance of the model on the validation set.
8. The method for enhancing and expanding magic box correlation training samples based on a small amount of labeled data according to claim 1, characterized in that: The specific steps of contrast learning sample enhancement in step 4 include: G1, contrastive learning application: G1-1, SimCSE framework: The unsupervised SimCSE model is used for sample enhancement. The embedding generated by Dropout2 times of the same sample is used as a positive sample, and combined with other samples in the batch as a negative sample pair; G1-2, loss function optimization: design a loss function to maximize the similarity of positive sample pairs embedding and minimize the similarity of negative sample pairs; G2, improves generalization ability: Enhance sample diversity: By comparing the samples generated by learning, we can increase the diversity and complexity of the training set and improve the model's ability to generalize to unseen data.
Citation Information
Patent Citations
Incremental learning method and system based on small number of labeled samples
CN112132179A
Sample information labeling method and device, equipment, program and storage medium
CN114676775A
Sample set optimization method and device, equipment, medium and product
CN115600109A
Text classification method and device, electronic equipment and storage medium
CN116628187A
Sample screening method and apparatus, and service object data searching method and apparatus
WO2019014607A1
Cited By
Data sample set generation method and model training method
CN120894789A
Sample data centralized optimization processing method applied to image recognition
CN121582701A
Dynamic prediction and resource allocation method for labeling task workload
CN121809996A
