Multi-task data screening method, device, equipment and medium based on large model
By constructing test sets, filter sets and splicing sets, using preset big models to calculate scores and filter data, the problem of data imbalance in large models in multi-task applications is solved, and data balance and equilibrium performance of large models are achieved.
CN118657214BActive Publication Date: 2025-03-25镁佳(北京)科技有限公司
2 Cites 0 Cited by
Patent Information
- Application Number
- CN202410624985.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-20
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2044-05-20
AI Technical Summary
Technical Problem
In multi-task applications, data imbalances are caused by the difficulty and cost of data acquisition of different tasks, which affects the model's balanced performance on different tasks.
Method used
By constructing test sets, filter sets and splicing sets, the test scores of each data are calculated using the preset big model, and excellent training data are filtered based on the score relationship, garbage data is filtered, and data of different tasks are balanced.
Benefits of technology
It effectively solves the problem of data quality and quantity imbalance in different tasks, improves the balanced performance of large models in multi-tasks, and reduces the training data requirements and costs.
✦ Generated by Eureka AI based on patent content.
Abstract
The present invention relates to the field of data processing technology, and discloses a multi-task data screening method, device, equipment and medium based on a large model, obtaining a data set corresponding to each task in the multi-task, wherein the data set includes a plurality of data; dividing the data set into a test set and a screening set, and obtaining a spliced set based on the test set and the screening set; using a preset large model to determine the first test score of each data in the test set and the second test score of each data in the spliced set; when the relationship between the second test score of any data in the spliced set and the first test score of the corresponding data in the test set meets a preset condition, the corresponding data in the spliced set is used as available data. This method can significantly reduce the amount of data required for training and save the cost of training a large model by filtering and screening data of different tasks and balancing the data of different tasks.
Need to check novelty before this filing date? Find Prior Art