Training data processing method and device and storage medium

Abstract

The invention provides a training data processing method and device and a storage medium, relates to the technical field of artificial intelligence, can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic and aided driving, and comprises the following steps: obtaining a pre-training regression model and an initial training set; obtaining a clustering result corresponding to each piece of sample data in the candidate data set by using a pre-training regression model; based on the initial training set and the second loss function, performing update constraint training on the pre-training regression model to obtain an intermediate model, and obtaining prediction confidence of each sample data in the first difference set by using the intermediate model; based on the prediction confidence and the clustering result, performing data sampling on sample data in the first difference set; updating the initial training set by using the obtained incremental training set; and performing loop iteration based on the updated initial training set, and taking the initial training set obtained when an iteration ending condition is satisfied as a target training set. The effectiveness of the training data is effectively improved.

Description

technical field [0001] The present application relates to the technical field of artificial intelligence, and in particular to a training data processing method, device and storage medium. Background technique [0002] A key issue that needs to be addressed in deep learning schemes is data availability. Usually, the accumulation of training data is based on the understanding and experience of technicians on the business to mark the sample data. This method relies on the knowledge of technicians to mark, which can easily lead to redundant training data. In addition, in general, there may be a large amount of data with high similarity in the sample data, and the data based on manual annotation will ignore the repetition of the data, which will damage the effectiveness of the training data and affect the training effect and application performance of related models. Therefore, it is necessary to provide a training data processing method capable of efficiently screening effecti...

AI Technical Summary

Problems solved by technology

Usually, the accumulation of training data is based on the technical staff’s understanding and experience of the business to label the sample data. This method relies on the technical staff’s knowledge to label, which can easily lead to redundant training data.

In addition, under normal circumstances, there may be a large amount of data with high similarity in the sample data, and the data based on manual annotation will ignore the repeatability of the data, which will damage the effectiveness of the training data and affect the training effect and application performance of related models.

Images