A neural network efficient training method based on sample representative adaptive evaluation
By constructing a gradient representativeness evaluation index during neural network training and dynamically adjusting the core sample subset, the problems of representativeness degradation and high computational overhead in existing technologies are solved, achieving efficient and stable model training results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2026-02-05
- Publication Date
- 2026-06-09
AI Technical Summary
Existing core sample subset training methods cannot dynamically identify representativeness degradation during neural network training, resulting in high computational overhead and easy overfitting problems. Furthermore, the reselection strategy relies on a fixed period and fails to effectively adapt to changes in model parameters.
We adopt an adaptive evaluation method based on sample representativeness. By constructing a gradient representativeness evaluation index, we dynamically adjust the core sample subset. This includes gradient matching optimization and greedy selection algorithms to construct the core sample subset. During training, we adaptively reselect the core sample subset based on the evaluation index, thereby reducing computational costs and maintaining model training stability.
It enables dynamic identification of representativeness degradation of core sample subsets during neural network training, reduces unnecessary computational overhead, suppresses training bias and overfitting, and improves training efficiency and accuracy. It is applicable to various deep neural network models and training scenarios.
Smart Images
Figure CN122174908A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer deep learning and model training acceleration technology, specifically to an efficient neural network training method based on adaptive evaluation of sample representativeness. Background Technology
[0002] In recent years, deep neural networks, represented by large language models, have made significant progress in various fields such as natural language processing, computer vision, and multimodal understanding. However, these models are typically characterized by large parameter sizes, massive amounts of training data, and high training computation costs, which limits their application scenarios where computing resources are limited or training efficiency is critical.
[0003] To reduce computational overhead during model training, coreset selection methods have gained increasing attention. These methods select a significantly smaller but representative subset of samples from the original training dataset and update the model parameters based on this subset, thereby reducing training costs while maintaining training effectiveness. Existing research has proposed various coreset construction methods based on gradient matching, submodule optimization, or greedy selection, and efficient solutions for coreset selection can be achieved through algorithms such as orthogonal matching pursuit.
[0004] However, existing core sample training methods still have shortcomings in practical applications. On the one hand, most methods reselect the core sample subset at fixed intervals or manually set frequencies during training, failing to dynamically adjust the reselection timing according to the model's training state, which can easily lead to unnecessary computational overhead. On the other hand, existing methods usually implicitly assume that the core sample subset maintains good representativeness across multiple training phases, failing to fully consider the fact that model parameters change continuously as training progresses. In actual training, due to changes in model parameter states, the ability of a fixed core sample subset to approximate the gradient of the overall training data may gradually decrease, causing the training direction to deviate from the overall data distribution, and even leading to overfitting problems.
[0005] Therefore, there is an urgent need for an efficient training method that can dynamically evaluate the fit between the core sample subset and the current model parameter state during neural network training, and adaptively decide whether to reselect the core sample subset accordingly. Summary of the Invention
[0006] To address the problems of existing core sample subset training methods, such as ineffective identification of core sample representativeness degradation, reliance on fixed periods for reselection strategies leading to low computational efficiency and high computational cost, this invention proposes a representativeness-aware driven dynamic core set training method. During neural network model training, this invention adaptively evaluates the representativeness of the core sample subset and dynamically adjusts the training sample subset accordingly. This method enables dynamic perception and adaptive control of the core sample subset's suitability, reducing training computational costs while ensuring the stability and accuracy of model training.
[0007] To achieve the above objectives, the present invention adopts the following technical solution.
[0008] An efficient neural network training method based on adaptive evaluation of sample representativeness uses a core sample subset to approximate the overall training data; it includes the following steps: 1) Construct a core sample subset and its corresponding weights for updating model parameters, and use the core sample subset to approximate the process of updating model parameters using the overall training data; 2) When the preset training and detection interval is reached, based on the statistical characteristics of the sample gradients in the core sample subset, calculate the gradient representativeness evaluation index to characterize the degree of adaptation of the current core sample subset to the gradient of the overall training data under the current model parameter state. 3) Compare the gradient representativeness evaluation index with a preset threshold. If the gradient representativeness evaluation index does not meet the threshold condition, it is determined that the current core sample subset is not representative enough, and the core sample subset reselection operation is triggered. If the gradient representativeness evaluation index meets the threshold condition, the current core sample subset is used to update the model parameters. 4) Repeat the above steps until the model training is complete.
[0009] In this invention, in step 1), during the core sample subset construction stage, subset selection of training samples is performed based on the gradient matching principle (that is, matching the overall gradient of the core set data with the overall gradient of the entire dataset); let the training dataset be... , No. The input features of each training sample are , No. The true label corresponding to each training sample is: The total number of samples in the training dataset is The model parameters are The model predicts the label as The single-sample loss function is The corresponding sample gradient is The average gradient of the entire training data is defined as: Given the core sample size constraint Under the given conditions, construct the following gradient matching optimization problem: ; Where S represents the selected core sample subset, and w represents the corresponding sample weight; By solving the gradient matching optimization problem described above, the weighted gradient of the core sample subset can be made to approximate the gradient direction of the entire training data in the gradient space.
[0010] In this invention, step 1) employs a greedy selection algorithm based on orthogonal matching pursuit (OMP) to solve the gradient matching optimization problem; specifically, the gradients of candidate samples are constructed into a gradient dictionary matrix: ; The gradient matching problem is then transformed into the following sparse linear approximation problem: ; The OMP selection process includes the following steps: ① Initialize the residual vector as follows The core sample set is initialized to empty; ② In each iteration, calculate the inner product between the current residual and the gradient of each candidate sample, and select the sample with the largest inner product value to add to the core sample set, that is: ; ③ Recalculate the sample weights using non-negative least squares on the selected sample set; ④ Update the residual vector and repeat the above steps until the preset core sample number is reached.
[0011] In this invention, in step 2), the gradient of a single sample in the core sample subset is treated as a random variable. Their expectations With variance They are defined as follows: ; Based on the above definition, a gradient signal-to-noise ratio (SNR) metric is constructed as an evaluation metric for the representativeness of the gradient in the core sample subset: .
[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention can dynamically identify the representativeness degradation problem of core sample subsets during neural network training, avoiding unnecessary computational overhead caused by fixed-period reselection; 2. This invention constructs representative evaluation indicators based on gradient statistics within a core sample subset, eliminating the need for frequent calculations of the gradient of the overall training data and significantly reducing evaluation costs; 3. Through an adaptive reselection mechanism driven by representativeness metrics, training bias and overfitting caused by outdated core sample subsets are effectively suppressed; 4. This invention does not limit the specific algorithm for constructing the core sample subset, exhibiting good versatility and applicability to various deep neural network models and training scenarios, including but not limited to Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Transformer models based on attention mechanisms, and their variants. Since the core sample subset construction and representativeness evaluation process of this invention relies solely on gradient information naturally generated during model training, without depending on specific network structures or task formats, it possesses good model independence and task versatility, adapting to deep neural network training scenarios of different scales and structures, and thus has strong practical value. Attached Figure Description
[0013] Figure 1 This diagram illustrates the changes in gradient difference and signal-to-noise ratio (SNR) with training epochs during the training process. (a) and (b) show the trends of gradient difference between the core sample subset and the overall training data with training epochs on the CIFAR-10 and CIFAR-100 datasets, respectively. (c) and (d) show the changes in the SNR calculated by gradient statistics of the core sample subset with training epochs on the corresponding datasets.
[0014] Figure 2 The diagram illustrates the comparison of classification performance and gradient difference for different core sample selection methods. (a) and (b) show the changes in model test accuracy with training epochs when different core sample selection methods are used for training on the CIFAR-10 and CIFAR-100 datasets, respectively. (c) and (d) show the changes in gradient difference generated by each method during training on the corresponding datasets.
[0015] Figure 3 A comparative diagram showing the combined effects of different core sample selection methods on training speedup and relative accuracy; (a) CIFAR-10; (b) CIFAR-100; (c) ImageNet. Detailed Implementation
[0016] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0017] This invention provides an efficient neural network training method based on adaptive evaluation of sample representativeness, comprising the following steps:
[0018] I. Construction of Core Sample Subset
[0019] In the process of training a neural network model, a core sample subset and its corresponding weights are first constructed for updating model parameters. The process of updating model parameters is approximated by the whole training data through the core sample subset.
[0020] In the core sample subset construction phase, subset selection of training samples is performed based on the gradient matching principle (that is, matching the overall gradient of the core set data with the overall gradient of the entire dataset); let the training dataset be... , No. The input features of each training sample are , No. The true label corresponding to each training sample is: The total number of samples in the training dataset is The model parameters are The model predicts the label as The single-sample loss function is The corresponding sample gradient is The average gradient of the entire training data is defined as: Given the core sample size constraint Under the given conditions, construct the following gradient matching optimization problem: ; Where S represents the selected core sample subset, and w represents the corresponding sample weight; by solving the above optimization problem, the weighted gradient of the core sample subset is made to approximate the gradient direction of the overall training data in the gradient space.
[0021] The above-described method for constructing the core sample subset is only one possible implementation, and this invention does not limit the specific algorithm for constructing the core sample subset. In one specific implementation, a core sample selection method based on the gradient matching principle can be used. Given a constraint on the number of core samples, the core sample subset and its corresponding weights are constructed by minimizing the difference between the overall gradient and the weighted gradient of the core sample subset. This optimization problem can be solved using greedy algorithms such as orthogonal matching pursuit.
[0022] When implementing this using a greedy selection algorithm based on Orthogonal Matching Pursuit (OMP), the gradients of the candidate samples are specifically constructed into a gradient dictionary matrix: ; The gradient matching problem is then transformed into the following sparse linear approximation problem: ; The OMP selection process includes the following steps: 1) Initialize the residual vector as follows The core sample set is initialized to empty; 2) In each iteration, calculate the inner product between the current residual and the gradient of each candidate sample, and select the sample with the largest inner product value to add to the core sample set, i.e.: ; 3) Recalculate the sample weights using non-negative least squares on the selected sample set; 4) Update the residual vector and repeat the above steps until the preset core sample number is reached.
[0023] II. Constructing Gradient Representativeness Evaluation Indicators
[0024] During training, based on the statistical characteristics of the gradients of the samples in the core sample subset, a gradient representativeness evaluation index is constructed to measure the degree of fit of the current core sample subset to the gradient of the overall training data under the current model parameter state.
[0025] 1) Quantitative description of the representativeness degradation phenomenon of core samples
[0026] During model training, the core sample subset is used to approximate the gradient update of the entire training data; however, as the model parameters change continuously, the ability of a fixed core sample subset to approximate the overall gradient gradually decreases. To quantify this phenomenon, this invention constructs a gradient mismatch metric.
[0027] In one implementation, let the training time be... The overall gradient estimate is The weighted gradient of the current core sample subset is The gradient direction mismatch index can be defined as: ; The directional difference between the overall gradient estimate and the weighted gradient of the core sample subset is defined to characterize the change in the representativeness of the core sample subset. When the gradient mismatch index increases with the training epochs, it indicates that the current core sample subset can no longer effectively represent the gradient characteristics of the overall training data in the current parameter space.
[0028] 2) Specific implementation methods of gradient representativeness evaluation indicators
[0029] To address the problem of representativeness degradation in core sample subsets, in one specific implementation, this invention constructs a representativeness evaluation index based on the gradient statistical characteristics of core sample subsets.
[0030] Treat the gradient of a single sample in the core sample subset as a random variable. Its expected value and variance are defined as follows:
[0031] Based on the above definition, the gradient signal-to-noise ratio metric is constructed as follows: ;
[0032] In practice, estimation is performed using the sample mean and sample variance of a core sample subset: ;
[0033] Under the assumption that the gradient satisfies the sub-Gaussian distribution condition, the upper bound of the approximate error of the core sample gradient mean to the true mean gradient is inversely proportional to the signal-to-noise ratio (SNR), thus enabling the SNR to serve as a criterion for measuring the representativeness of the gradient of the core sample subset.
[0034] In practical calculations, the gradients of the core sample subset are statistically analyzed, and the sample mean and sample variance are used to estimate the above-mentioned index. This gradient signal-to-noise ratio index reflects the relationship between the average gradient signal strength and gradient dispersion of the core sample subset, and is thus used to evaluate the degree to which the core sample subset fits the gradient of the overall training data.
[0035] III. Adaptive Reselection Based on Representativeness Assessment
[0036] When the preset training and detection interval is reached, the gradient representativeness evaluation index is compared with the preset threshold. If the evaluation index does not meet the threshold condition, it is determined that the current core sample subset is not representative enough, and the core sample subset reselection operation is triggered. If the evaluation index meets the threshold condition, the current core sample subset is used to update the model parameters.
[0037] The algorithm is shown in Table 1:
[0038] Table 1
[0039]
[0040] During model training, the present invention calculates the gradient representativeness evaluation index for the current core sample subset according to a preset detection interval and compares it with a preset threshold.
[0041] When the evaluation metric is below the threshold, the current core sample subset is deemed insufficiently representative, triggering a core sample subset reselection operation to reconstruct the core sample subset and its weights; when the evaluation metric is not below the threshold, the current core sample subset is used to update the model parameters.
[0042] Through the above process, this invention achieves dynamic perception and adaptive control of the representativeness degradation of the core sample subset. While reducing the computational overhead of unnecessary sample reselection, it ensures the stability and effectiveness of gradient update direction during model training, achieving training accuracy while improving overall training efficiency.
[0043] In summary, this invention embeds the aforementioned core sample subset construction, representativeness evaluation, and adaptive reselection mechanism into the neural network model training process, forming a dynamic iterative and efficient training process. Specific embodiments are described below.
[0044] Example 1
[0045] In this experiment, CIFAR-10, CIFAR-100 and ImageNet datasets were selected as experimental subjects. ResNet-18 was used as the basic neural network model for image classification tasks. The training effects of different core sample selection methods under limited data budget conditions were compared and evaluated.
[0046] The data budget represents the proportion of training samples used to construct the core sample subset, set to 5%, 10%, and 20% of the original training data size, respectively. Under the same network structure, optimization algorithm, and training epochs, the classification accuracy of models obtained by different methods on the test set is evaluated.
[0047] Figure 1 This demonstrates how the gradient difference (GDiff) and signal-to-noise ratio (SNR) change with training epochs under different dataset conditions during model training. Figure 1 (a) and Figure 1 (b) shows the gradient difference between the core sample subset and the overall training data on the CIFAR-10 and CIFAR-100 datasets as a function of training rounds. Figure 1 (c) and Figure 1 (d) represents the change in the signal-to-noise ratio (SNR) calculated by gradient statistics of the core sample subset on the corresponding dataset with each training epoch. Figure 1 It can be seen that as training progresses, the gradient difference index generally shows an increasing trend in the several rounds of training after selecting the core set, while the signal-to-noise ratio index gradually decreases. This indicates that under the condition of a fixed core sample subset, its representativeness of the overall training data will gradually degrade with the training process, thus providing a basis for the present invention to propose a dynamic identification of the representativeness degradation problem of the core sample subset.
[0048] The gradient mismatch is mitigated as shown in Table 2.
[0049]
[0050] In Table 2, the Random, Craig, Glister, and GradMatch methods reselect the core sample subset every 20 training epochs; the CREST method determines the update timing of the core sample subset based on its quadratic loss approximation; and the proposed method ERACS dynamically updates the core sample subset based on the signal-to-noise ratio (SNR) metric. "Full" indicates the result of training with the complete training data. As shown in Table 2, under different datasets and data budgets, the proposed method ERACS achieves high test accuracy even with limited training sample size. Especially on the more complex CIFAR-100 and ImageNet datasets, as the training process progresses, the ERACS method effectively alleviates the training performance degradation caused by fixed subsets or fixed update cycles by dynamically identifying and updating the representative degraded core sample subset based on the SNR metric, thus maintaining high model accuracy while ensuring training efficiency. These results demonstrate that the proposed method exhibits good stability and versatility under different dataset sizes and complexities, effectively maintaining model performance while reducing the training data size.
[0051] Figure 2 The results show a comparison of the training performance of the method of this invention with existing core sample selection methods on different datasets. Among them, the results are provided by... Figure 2 As can be seen, compared with the GradMatch method, the method of this invention can effectively suppress the drastic fluctuations of gradient differences during training while maintaining the model testing accuracy, thereby mitigating the training instability and performance degradation caused by the staleness of the core sample subset.
[0052] Figure 3 This paper presents a comparison of training speedup and relative accuracy for different core sample selection methods on the CIFAR-10, CIFAR-100, and ImageNet datasets. Training speedup measures the computational efficiency improvement of training with a subset of core samples compared to training with the full training data, while relative accuracy measures the model's performance retention when the number of training samples is reduced. Figure 3 It can be seen that the method of the present invention can significantly improve training efficiency while maintaining model accuracy similar to that under complete training data conditions under various dataset conditions, indicating that the method of the present invention has good practicality and versatility in deep learning tasks of different scales and complexities.
Claims
1. A neural network efficient training method based on sample representative adaptive evaluation, characterized in that, Approximate modeling of the overall training data is performed using a core subset of samples; this includes the following steps: 1) Construct a core sample subset and its corresponding weights for updating model parameters, and use the core sample subset to approximate the process of updating model parameters using the overall training data; 2) When the preset training and detection interval is reached, based on the statistical characteristics of the sample gradients in the core sample subset, calculate the gradient representativeness evaluation index to characterize the degree of adaptation of the current core sample subset to the gradient of the overall training data under the current model parameter state. 3) Compare the gradient representativeness evaluation index with a preset threshold. If the gradient representativeness evaluation index does not meet the threshold condition, it is determined that the current core sample subset is not representative enough, and the core sample subset reselection operation is triggered. If the gradient representativeness evaluation index meets the threshold condition, the current core sample subset is used to update the model parameters. 4) Repeat the above steps until the model training is complete.
2. The neural network training method of claim 1, wherein, In step 1), during the core sample subset construction stage, subset selection of training samples is performed based on the gradient matching principle.
3. The method of claim 2, wherein the set of parameters is determined by a neural network. The training dataset is , For the first Input features of each training sample, For the first The true labels corresponding to each training sample are given, and the total number of samples in the training dataset is given. The model parameters are The single-sample loss function is The corresponding sample gradient is The average gradient of the entire training data is defined as: Given the constraint m on the number of core samples, construct the following gradient matching optimization problem: ; Where S represents the selected core sample subset, and w represents the corresponding sample weight; By solving the gradient matching optimization problem described above, the weighted gradient of the core sample subset can be made to approximate the gradient direction of the entire training data in the gradient space.
4. The efficient neural network training method according to claim 3, characterized in that, A greedy selection algorithm based on orthogonal matching pursuit (OMP) is used to solve the gradient matching optimization problem.
5. The efficient neural network training method according to claim 4, characterized in that, The following method is used to solve the gradient matching optimization problem using a greedy selection algorithm based on orthogonal matching pursuit (OMP): Construct a gradient dictionary matrix from the gradients of the candidate samples: ; The gradient matching problem is then transformed into the following sparse linear approximation problem: ; The OMP selection process includes the following steps: ① Initialize the residual vector as follows The core sample set is initialized to empty; ② In each iteration, calculate the inner product between the current residual and the gradient of each candidate sample, and select the sample with the largest inner product value to add to the core sample set, that is: ; ③ Recalculate the sample weights using non-negative least squares on the selected sample set; ④ Update the residual vector and repeat the above steps until the preset core sample number is reached.
6. The efficient neural network training method according to claim 1, characterized in that, In step 2), the gradient of a single sample in the core sample subset is treated as a random variable. Their expectations With variance They are defined as follows: ; Based on the above definition, a gradient signal-to-noise ratio (SNR) metric is constructed as an evaluation metric for the representativeness of the gradient in the core sample subset: 。