An efficient dynamic dataset pruning method and system based on an evaluation model

Abstract

The application discloses a kind of efficient dynamic data set pruning method and system based on evaluation model, it is related to artificial intelligence, present technique in view of the problem that pruning effect is not ideal proposes the scheme of the present application.The evaluation model is used to assist the generation of pruning index, and a sample training strategy is designed to reduce the information loss caused by pruning.The advantages are that, on the one hand, the evaluation model is used to extract the features of the data set in advance, providing more accurate indicators for data pruning;On the other hand, the online batch evaluation method is used to effectively solve the lag problem of traditional dynamic indicators.Through these two innovative means, the efficiency and accuracy of data pruning can be significantly improved, and the training process is optimized.The sample training strategy proposed in the application solves the problem of data set calling order, reduces unnecessary computational complexity, thereby improving the training efficiency and enhancing the generalization ability of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence, and particularly relates to an efficient dynamic dataset pruning method and system based on an evaluation model. BACKGROUND

[0002] In recent years, a large number of studies have shown that large-scale datasets can significantly improve the performance of deep learning models, which has driven the construction and application of more and more large-scale datasets. However, as the size of the data continues to expand, the performance improvement also shows a marginal diminishing effect. In particular, for extremely large datasets, to achieve a slight performance improvement, it may require orders of magnitude more data. However, it is not easy to handle such a large amount of data. Especially when training image or video data with high redundancy and strong pixel correlation, the demand for computing resources and time increases dramatically, bringing great challenges to practical applications. In order to address this problem, researchers have invested a lot of effort to explore how to accelerate model training. A direct and effective acceleration method is dataset pruning, which aims to filter out a high-quality subset that can maintain comparable generalization ability to the complete dataset, or reduce the calling frequency of the complete dataset, thereby achieving acceleration of the training process.

[0003] However, the current dataset pruning method still has many defects and deficiencies. Specifically, the static dataset pruning method (i.e. core dataset selection) is often prone to bias in the distribution of dataset features in practice. Since the static method usually selects a subset with fixed rules or indicators when pruning, it ignores the dynamics and diversity of data distribution, and this bias will further intensify as the pruning ratio increases. Especially in the case of high pruning rate, the subset may not be able to fully cover the key features of the complete dataset, resulting in a significant decline in the training effect of the model, and even a catastrophic collapse of performance. This problem is particularly significant in scenarios where deep learning models are sensitive to data distribution, such as when dealing with data with high-dimensional features or complex relationships, static pruning methods are difficult to effectively retain the diversity and representativeness of the data.

[0004] To address the above problems, some methods have been proposed. Qin et al. in "Qin Z, Wamg K, Zheng Z, et al. InfoBatch: Lossless Training Speed Up by Unbiased Dynamic Data Pruning. [C]. In Proceedings of the International Conference on Learning Representations, 2024" proposed a method that uses loss value as a pruning indicator and introduces unbiased pruning and weighting strategies to accelerate the training process, aiming to reduce training time while maintaining model performance. However, there are still some problems that need to be solved in existing methods.

[0005] Firstly, the inevitable information loss and bias in the pruning process is still a key challenge. Although using loss value as the basis for pruning can better preserve data that contributes to model training, in the case of high pruning rate, some important training samples may be ignored, leading to performance degradation of the model when dealing with complex data, and even bias in learning certain key features.

[0006] Secondly, the pruning indicator in dynamic pruning is not accurate and has a lag. Although dynamic pruning methods attempt to dynamically adjust pruning strategies based on real-time feedback during training, due to the selection and calculation lag of the pruning indicator, the pruning operation may not timely reflect the latest changes in data distribution. This lag makes the pruning decision unable to fully capture the time-varying characteristics of the data, affecting the accuracy of the pruning effect and the efficiency of the training process.

[0007] In addition, in the early stages of model training, the convergence direction of the model is not clear and has a lot of randomness, which also makes the loss value not effective in guiding data pruning at this stage. When the model is first trained, the parameters and gradients change greatly, and the model's dependence on data has not stabilized, and the loss value fluctuates sharply. Therefore, simply relying on loss value to determine which data can be pruned is unreliable, which may lead to some potentially important data being incorrectly removed, affecting the training process and final results of the model. SUMMARY

[0008] The present application aims to provide an efficient dynamic data set pruning method and system based on an evaluation model to solve the problems existing in the prior art.

[0009] The efficient dynamic data set pruning method based on an evaluation model in the present application uses an evaluation model to assist in generating pruning indicators and designs a sample training strategy to reduce information loss caused by pruning.

[0010] The evaluation model training includes the following steps:

[0011] S1. According to the size and complexity of the data set, adjust the depth and width of the network; the output structure of the last layer of the evaluation model is defined as a single data difficulty index;

[0012] S2. Define the loss function of the evaluation model

[0013] Wherein, x is the input data, y is the label corresponding to the data, loss p (x,y) represents the cross-entropy loss from the pre-trained classification model, H(x) represents the output of the evaluation model, and theta represents the parameters of the evaluation model; the loss of the pre-trained model is taken as the training target, so that H(x) and loss p (x,y) present a positive correlation;

[0014] S3. The evaluation model is used as a plug-and-play module, and is applied to the data pruning process in an online batch data selection manner;

[0015] The sample training strategy design steps are as follows:

[0016] S4. In this training strategy, for relatively simple data sets, samples with higher difficulty are put into the early training period to enable the model to quickly adapt and capture complex features in the data; and in more complex data sets, samples with relatively low difficulty are used first to ensure that the model converges stably in the early stage, laying a solid foundation for subsequent processing of more complex samples.

[0017] The efficient dynamic data set pruning system based on the evaluation model in the application utilizes the method for pruning processing.

[0018] The efficient dynamic data set pruning method and system based on the evaluation model in the application have the advantages that, on the one hand, the evaluation model extracts data set features in advance to provide more accurate indicators for data pruning; on the other hand, the online batch evaluation method effectively solves the lag problem existing in traditional dynamic indicators. Through these two innovative means, the application can significantly improve the efficiency and accuracy of data pruning and optimize the training process. The sample training strategy proposed in the application solves the problem of data set calling order, reduces unnecessary calculation amount, thereby improving the training efficiency and enhancing the generalization ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0019] Fig. 1 is a pre-training process schematic diagram of the evaluation model in the application.

[0020] Fig. 2is a flowchart of the pruning method described in the present application.

[0021] Fig. 3 is a structural diagram of the CIFAR dataset evaluation model in the present application.

[0022] Fig. 4 is a structural diagram of the ImageNet dataset evaluation model in the present application. DETAILED DESCRIPTION

[0023] The core of the efficient dynamic dataset pruning method based on evaluation model in the present application lies in introducing a lightweight model as an evaluation tool for pruning, so as to fully exploit the potential knowledge of the pre-trained model, thereby significantly improving the performance of dataset pruning. This method uses a lightweight evaluation model to comprehensively analyze and model the features of the dataset, and extracts key information of the data distribution through the pre-training stage, providing accurate and efficient indicators for pruning. Compared with traditional pruning methods, this scheme can better capture the importance and diversity of data, and avoid information loss and bias caused by improper pruning. At the same time, since the lightweight model has low computational complexity, this method can quickly evaluate and make pruning decisions on large-scale datasets without significantly increasing the training cost. This innovative method not only improves the efficiency and reliability of the pruning process, but also enhances its adaptability and universality, suitable for various deep learning tasks and scenarios. By combining lightweight evaluation models with data pruning, the present application provides a new solution for model training acceleration and resource optimization, which has important application value and practical significance.

[0024] As shown in Figs. 1 to 4 , the efficient dynamic dataset pruning method based on evaluation model in the present application includes two parts: evaluation model training and sample training strategy.

[0025] The evaluation model is used to assist in generating pruning indicators, and the evaluation model training specifically includes the following steps:

[0026] S1. Evaluation model structure design: According to the size and complexity of the dataset, adjust the depth and width of the network. The output structure of the last layer of the evaluation model is defined as a single data difficulty indicator, i.e. the model evaluates the difficulty of each sample according to this indicator. This design significantly simplifies the evaluation task, enabling the evaluation model to quickly and accurately determine which data samples are crucial to the training process and which may be eliminated in the pruning process. In this way, the evaluation model can more efficiently guide the data pruning, while keeping the model structure lightweight, avoiding unnecessary complex calculations and resource waste.

[0027] S2. Pre-training: The output value of the evaluation model represents the difficulty level of the corresponding sample. A pre-trained model is used to guide the learning of the evaluation model, so that the evaluation model can better learn the characteristics of the data. Define the loss function of the evaluation model

[0028] where x is the input data, y is the label of the corresponding data, loss p (x,y) represents the cross-entropy loss from the pre-trained classification model, H(x) represents the output of the evaluation model, and θ represents the parameters of the evaluation model. The fraction , which means the result of the softmax transformation of the input H(x). The main goal is to enable the lightweight evaluation model to simulate the judgment of the large model in evaluating the difficulty of the data. To achieve this goal, the loss of the pre-trained model is used as the training target to promote the positive correlation between H(x) and loss p (x,y). This optimization target ensures that the evaluation network can be consistent with the sample difficulty based on the data characteristics. For each data point, the evaluation model outputs a scalar value H(x). For difficult samples corresponding to higher loss p (x,y), H(x) will increase as the optimization process progresses. In addition, the softmax function in the loss formula amplifies the differences between data points, enhancing the model's ability to distinguish and prioritize difficult samples. This mechanism ensures that the evaluation network can effectively capture the relative difficulty of each data point, thereby promoting a more refined pruning process.

[0029] Pre-training specifically includes the following sub-steps:

[0030] S21. The input picture x is simultaneously transmitted to the pre-trained evaluation model and the teacher model. The goal of the evaluation model is to predict the difficulty of the picture, while the teacher model outputs the class prediction or other features used for training of the picture.

[0031] S22. The evaluation model outputs a scalar value representing the difficulty of the data. At the same time, the teacher model generates a class prediction output M(x). By comparing with the true label y, the cross-entropy loss value of the teacher model is calculated: loss p (x,y) = -∑ (x,y)∈D ylog(M(x));

[0032] S23. The output of the evaluation model batch data is normalized by softmax to ensure that the output value is within the range [0, 1] and conforms to the probability distribution. Then, according to formula, the loss value of the evaluation model is calculated, which is the cross-entropy loss between the evaluation model and the teacher model result, reflecting the performance of the evaluation model in capturing the difficulty of the data.

[0033] S24. Finally, the parameters θ of the evaluation model are optimized using the gradient descent algorithm. The output of the evaluation model will be closer to the output of the teacher model, thus improving the evaluation model's accuracy in judging the difficulty of the data. In this way, the evaluation model gradually adjusts its parameters so that its output better reflects the difficulty of the data samples, ultimately providing effective guidance for data pruning. The output of the evaluation model will be closer to the output of the teacher model, thus improving the evaluation model's accuracy in judging the difficulty of the data. In this way, the evaluation model gradually adjusts its parameters so that its output better reflects the difficulty of the data samples, ultimately providing effective guidance for data pruning.

[0034] S3. The evaluation model is used as a plug-and-play module in the data pruning process, thus accelerating the training. In this process, an online batch data selection method is used. Specifically, a batch of data is first input into the evaluation model, which quickly generates evaluation indicators for each sample. Based on these indicators, the samples are sorted and the most representative and important ones are selected, thus reducing the amount of data used in the model training process. Through this online batch selection method, the amount of training data can be greatly reduced without sacrificing model performance, thus accelerating the model training process. This method not only improves data utilization efficiency, but also optimizes the allocation of computing resources, making the training process more efficient and flexible.

[0035] The sample training strategy is designed to reduce the information loss caused by pruning, with the following specific steps:

[0036] S4. The training strategy is specified by analyzing the key stages of model training and the characteristics of the data set. The analysis process is as follows: first, the early stage of training is a critical stage for establishing the association between data and model, and introducing pruning interference in the early stage of training will have a greater impact than introducing pruning interference in the late stage of training, which determines the performance and learning effect of the model in subsequent training; second, for simple data sets, giving more weight to difficult samples can help the model training, and the priority of difficult features or difficult samples is usually higher than that of simple features or simple samples; for more difficult or larger data sets, the priority of simple features is usually higher than that of difficult features. Therefore, the implementation method of this training strategy is as follows: in simple data sets, difficult samples are preferentially retained in the early stage of training to enable the model to quickly adapt to and capture complex features in the data; for complex data sets, simple samples are preferentially placed in the early stage of training to ensure that the model converges stably in the early stage and lays a good foundation for processing complex samples in the later stage. In different types of data sets, the priority of features and samples needs to be dynamically adjusted according to the complexity of the data. This training strategy can dynamically adjust the training order of samples according to the characteristics of the data set, thus improving the efficiency and effectiveness of model training.

[0037] The efficient dynamic data set pruning system based on the evaluation model in the present application uses the method for pruning processing.

[0038] For those skilled in the art, other various corresponding changes and modifications can be made according to the technical solutions and concepts described above, and all these changes and modifications shall belong to the protection scope of the claims of the present application.

Claims

1. An efficient dynamic dataset pruning method based on an evaluation model, characterized in that, For image classification, the model is evaluated to help generate pruning metrics and a sample training strategy is designed to reduce information loss caused by pruning. The training of the evaluation model includes the following steps: S1. Adjust the depth and width of the network according to the size and complexity of the dataset; evaluate the output structure of the last layer of the model by defining a single data difficulty metric. S2. Define the loss function for evaluating the model. ; in, Let be the input image data, and y be the corresponding ground truth label. This represents the cross-entropy loss from the pre-trained classification model. This represents the output of the evaluation model. This represents the parameters of the evaluation model; the loss of the pre-trained model is used as the training objective to encourage... and There is a positive correlation between them; S3. By pre-training, a correlation is established between the data and the evaluation model. The evaluation model is then used as a plug-and-play module and applied to the data pruning process using online batch data selection. The sample training strategy design includes the following steps: S4. A method is adopted to place important data in the key stage for training. The early stage of training is the key stage of model training. In simple datasets, difficult samples are reserved in the early stage of training first, so that the model can quickly adapt and capture the complex features in the data. For complex datasets, simple samples are put into the early stage of training first, so as to ensure that the model can converge stably in the early stage and lay a good foundation for processing complex samples later. Step S2 specifically includes the following sub-steps: S21. Input image The data is simultaneously passed to both the pre-trained evaluation model and the teacher model; the evaluation model aims to predict the difficulty of the image, while the teacher model outputs the image's category prediction or relevant features used for training. S22. The evaluation model outputs a scalar value regarding the difficulty of the data; simultaneously, the teacher model generates a category prediction output. ; By comparing with real labels Compare and calculate the cross-entropy loss value of the teacher model: ; S23. Perform softmax normalization on the output of the batch data of the evaluation model to ensure that the output value is within the range of [0,1] and conforms to the probability distribution; then according to The formula calculates the loss value of the evaluation model, which is the cross-entropy loss between the evaluation model and the teacher model results, reflecting the evaluation model's performance in terms of the difficulty of capturing data; S24. Use the gradient descent algorithm to optimize the parameters of the evaluation model. .

2. The efficient dynamic dataset pruning method based on an evaluation model according to claim 1, characterized in that, Specifically, step S3 involves: a batch of data is first input into the evaluation model, which quickly generates evaluation metrics for each sample; the samples are sorted according to these metrics, and then the most representative and important samples are selected according to step S4, thereby reducing the amount of data used in the model training process.

3. An efficient dynamic dataset pruning system based on an evaluation model, characterized in that, Pruning is performed using the method described in any one of claims 1-2.

Images