A deep neural network training acceleration method and system based on a sample priority mechanism, an electronic device, and a storage medium

Abstract

A method, system, electronic device, and storage medium for accelerating deep neural network training based on a sample priority mechanism are disclosed. The method includes: performing a first forward propagation on all samples in the current training batch, calculating the loss value of each sample, and obtaining its corresponding class label; grouping samples by class within the batch based on the loss value and class label, and within each class, selecting high-loss samples according to a preset class retention ratio or quota based on the loss value, and merging them to form a backpropagation dataset; performing a second forward and backpropagation only on samples in the backpropagation dataset to calculate gradients and update model parameters; and skipping the backpropagation process for samples not selected in the backpropagation dataset. This invention improves the convergence stability and final generalization performance of the model.

Description

Technical Field

[0001] This disclosure belongs to the field of artificial intelligence technology, specifically relating to a method, system, electronic device, and storage medium for accelerating deep neural network training based on a sample priority mechanism. Background Technology

[0002] In current deep neural network training practices, mini-batch stochastic gradient descent and its variants are widely used, with all samples in each batch participating equally in the complete forward and backward propagation calculations. However, as model training progresses, the contribution of a large number of simple or well-learned samples to gradient updates tends to saturate. Continuing to perform computationally intensive backpropagation on these samples not only yields low marginal returns but also results in significant waste of computational resources and prolonged training time. To address this issue, existing techniques have proposed methods such as importance sampling, curriculum learning, and loss- or gradient-based sample selection. However, these methods have limitations in terms of efficiency and stability: importance sampling requires maintaining the historical state of all samples, leading to additional memory and management overhead; curriculum learning primarily focuses on final performance improvement rather than training acceleration and may increase process complexity; while simple loss-threshold-based selection methods, although caching is unnecessary, are prone to missing minority class samples due to neglecting class balance, introducing training instability and model bias, making it difficult to achieve stable and efficient training acceleration while maintaining convergence quality.

[0003] Invention / Utility Model Content

[0004] The first aspect of this disclosure provides a method for accelerating deep neural network training based on a sample priority mechanism, comprising the following steps:

[0005] Step 1: Perform the first forward propagation on all samples in the current training batch, calculate the loss value for each sample, and obtain its corresponding class label;

[0006] Step 2: Based on the loss value and category label, group the samples by category within the batch, and within each category, select high-loss samples according to the loss value and a preset category retention ratio or quota, and merge them to form a backpropagation dataset;

[0007] Step 3: Perform a second forward and backward propagation only on samples in the backpropagation dataset to calculate gradients and update model parameters; for samples not selected in the backpropagation dataset, skip their backpropagation process.

[0008] Preferably, the step of grouping samples by category within a batch based on the loss value and category label, and then selecting high-loss samples within each category according to a preset category retention ratio or quota based on the loss value, specifically includes:

[0009] Divide the sample index within a batch into multiple category subsets based on the category labels;

[0010] For each category subset, calculate the number of samples to be selected based on the preset retention ratio for that category, or directly use the preset quota;

[0011] Within each category subset, the top k sample indices are selected in descending order of loss values, where k is the calculated number of samples or a preset quota;

[0012] The sample indices selected from all category subsets are merged to obtain the global selected sample set, and the backpropagation dataset is constructed accordingly.

[0013] Preferably, the category retention ratio or quota is a fixed value set in advance, or a variable value that is dynamically adjusted during the training process according to the training stage, model status, or data distribution.

[0014] Preferably, in step three, an independent computation graph is constructed for the backpropagation dataset by performing a second forward propagation, so that backpropagation is performed only on the reduced computation graph, thereby reducing the tensor size and computational load involved in backpropagation.

[0015] Preferably, the method is applicable to deep learning frameworks that employ dynamic graph mechanisms. By constructing a sub-computation graph through the second forward propagation, the problem that zeroing out the loss of unselected samples cannot substantially reduce computational overhead is avoided.

[0016] Preferably, the method is integrated into the training iteration of standard stochastic gradient descent or its variants, and is compatible with different optimizers, learning rate adjustment strategies, and distributed training frameworks.

[0017] Secondly, the present invention provides a deep neural network training acceleration system based on a sample priority mechanism, comprising:

[0018] The evaluation module performs the first forward propagation on the current training batch to calculate the loss value for each sample and obtain its class label;

[0019] A filtering module, connected to the evaluation module, is used to receive the loss value and class label, and perform high-loss sample filtering by class grouping to construct a backpropagation dataset;

[0020] The training update module, connected to the filtering module, is used to perform a second forward and backward propagation only on the backpropagation dataset to update the model parameters and skip the backpropagation of non-selected samples.

[0021] Preferably, the filtering module further includes:

[0022] Grouping units are used to divide batch samples into multiple category subsets based on category labels;

[0023] Quota determination unit, used to determine the number of samples to be selected for each category subset;

[0024] The selection unit is used to select a specified number of high-loss sample indices within each category subset, sorted by loss value.

[0025] The merging unit is used to merge the sample indices selected from all category subsets to form the backpropagation dataset.

[0026] Thirdly, the present invention provides an electronic device, comprising:

[0027] At least one processor;

[0028] And, a memory communicatively connected to the at least one processor;

[0029] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in any one of the first aspects of the present invention.

[0030] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the first aspects of the present invention.

[0031] The solution of this invention achieves technological breakthroughs and performance improvements at multiple levels by introducing a core mechanism that combines "high-loss sample screening by category grouping" with "secondary forward propagation to construct a sub-computation graph". First, at the algorithm level, this mechanism prioritizes high-loss samples that contribute more to the convergence of the current model while ensuring the diversity and category coverage of gradient information through category quota constraints. This effectively avoids training instability and model bias caused by the absence of minority class samples, thus significantly improving the model's convergence stability and final generalization performance while accelerating training. Second, in terms of computational efficiency, by performing independent secondary forward propagation on the selected key sample subset, a lightweight computation graph covering only the selected samples can be constructed in dynamic graph frameworks such as PyTorch. This fundamentally reduces the tensor size and computational path required for backpropagation, achieving a substantial reduction in backpropagation overhead and obtaining a verifiable and significant real training speedup. Finally, in terms of system applicability, this method performs dynamic filtering at the batch level, without the need to maintain a global historical state. The parameters are flexibly configured and can be seamlessly integrated into existing optimizers and distributed training frameworks. It achieves intelligent reallocation of computing resources in a low-intrusive manner and demonstrates excellent acceleration and robustness across a wide range of tasks and data distributions. Attached Figure Description

[0032] Figure 1 It is a deep neural network training acceleration algorithm based on a sample priority mechanism;

[0033] Figure 2 This is a typical experimental training process diagram (model ViT-B / 16, dataset CIFAR-10);

[0034] Figure 3 This is a typical experimental performance time plot (model ViT-B / 16, dataset CIFAR-10);

[0035] Figure 4 This is a flowchart of a deep neural network training acceleration method based on a sample priority mechanism according to the present invention;

[0036] Figure 5 This is a flowchart of the present invention, which describes how, based on the loss value and category label, samples are grouped by category within a batch, and within each category, high-loss samples are selected according to a preset category retention ratio or quota based on the magnitude of the loss value.

[0037] Figure 6 This is a block diagram of the deep neural network training acceleration system of the present invention;

[0038] Figure 7 This is a system block diagram of the screening module of the present invention. Detailed Implementation

[0039] This invention discloses a method for accelerating deep neural network training based on a sample priority mechanism, and provides a corresponding deep neural network training acceleration system and computer-readable storage medium. The sample priority mechanism refers to: grouping samples within a batch according to category labels, using the magnitude of the sample loss value to characterize the "importance" of a sample to the convergence of the current model, and prioritizing samples with higher losses within each category according to a preset retention ratio (or quota), forming a set of selected samples that balances "high loss priority" and "category balance". For this set of important selected samples, a second forward propagation is used to reconstruct a computational graph covering only the selected samples, and backpropagation and parameter updates are performed only on this set; for "simple samples" that are not selected, only their loss evaluation results from the first forward propagation are retained, and the backpropagation process is skipped. By employing the aforementioned priority training strategy of "high-loss priority screening by category grouping + secondary forward and backward propagation only for selected samples", this invention can ensure that category information fully participates in training, improves convergence stability and convergence quality, while concentrating the main computing resources on samples that contribute more to the improvement of model performance. Thus, without significantly sacrificing model performance, it can significantly shorten the training time of deep neural networks and reduce training computational overhead.

[0040] In deep neural network training, standard mini-batch stochastic gradient descent typically performs forward and backward propagation on all samples within each batch. Backpropagation is the main computational bottleneck, especially in scenarios involving large models and large-scale data training, where it incurs significant time and computational overhead. Furthermore, the "effective contribution" of training samples to model parameter updates is uneven: many "simple samples" (low loss, already fitted by the model) have limited marginal contributions to gradient updates, yet they are still subjected to repeated, expensive backpropagation, resulting in wasted computational resources. Moreover, in real-world tasks, training data often exhibits class imbalance or a long-tailed distribution. Minority class samples are few in number but crucial for the model's discrimination boundary and generalization performance. If random sampling or selective learning is used solely based on loss thresholds / loss ranking, some class samples are easily discarded excessively or repeatedly missing during the selection process, leading to insufficient class coverage of gradient information. This introduces class bias, exacerbates performance degradation caused by imbalance, and can even cause training oscillations and convergence instability. Existing learning strategies either struggle to balance the acceleration goal of "reducing backpropagation computation" with the performance goal of "maintaining class coverage / convergence quality," or require additional complex distribution statistics and parameter tuning costs, making it difficult to achieve a good balance between training acceleration, accuracy preservation, and convergence stability.

[0041] Next, with reference to the accompanying drawings, the method of the present invention will be described, wherein Figure 1 It is a deep neural network training acceleration algorithm based on a sample priority mechanism; Figure 2 This is a typical experimental training process diagram (model ViT-B / 16, dataset CIFAR-10); Figure 3 This is a typical experimental performance timeline (model ViT-B / 16, dataset CIFAR-10). The deep neural network training acceleration method of this invention includes the following steps:

[0042] Step 1: Assess the importance of the current batch and obtain category information.

[0043] In a training iteration, the current batch of data is input. Where x i Let y be the input for the i-th sample. i For its monitoring signal, c i Assign category labels (for category constraint filtering). For the current batch In model f θ Perform a forward propagation once to calculate the sample-by-sample loss:

[0044] l i =l(f θ (x i ),y i (1)

[0045] This forward process is only used to assess sample importance and provide a basis for screening, without triggering parameter updates, thus avoiding the need to perform costly backpropagation calculations on the entire batch.

[0046] Step 2: Introduce a high-loss sample screening mechanism grouped by category to construct the backpropagation dataset.

[0047] After obtaining the sample-by-sample loss With category labels Next, high-loss sample filtering by category is performed within the batch. First, the sample indices are grouped according to the category label to obtain the index set corresponding to each category:

[0048]

[0049] Then, a preset retention ratio ρ is set for each category k. k ∈(0,1](or equivalently pre-set quota q for each class) k ), and calculate the number of samples selected for that category. Within the category, the loss l i Using priority metrics for sorting and selection, from the set Select the top q with the largest loss k Each sample index constitutes I k And perform Top-k selection on each category set:

[0050]

[0051] The selected index sets of each category are merged to obtain the global selected set S = ∪ k I k Based on this, a backpropagation dataset was constructed. This method biases the screening results towards high-loss samples to enhance their contribution to the current convergence, while also being influenced by p in the category dimension. k (or q) k Constraints are used to avoid class concentration, missing classes, or class imbalance caused by filtering solely by loss, ensuring that gradient information maintains class coverage and diversity, thereby improving convergence stability and convergence quality.

[0052] Step 3: Perform a second forward and backward propagation update on the backpropagation dataset.

[0053] Only for the backpropagation dataset obtained in step two A second forward propagation is performed to reconstruct a computation graph covering only the selected samples, and backpropagation and parameter updates are then performed on this computation graph. Specifically, the training objective is computed and updated on the set of selected samples. After obtaining the set of selected indices S, the training objective is computed and parameters are updated only on the selected set, and the gradients of the model parameters are calculated.

[0054]

[0055] Finally, the optimizer updates θ←Opt(θ,g). For samples that are not selected...

[0056] Only the loss evaluation results generated in step one are retained, and the backpropagation process is skipped, excluding them from gradient calculation and parameter updates. This concentrates the main computational power on samples that contribute more significantly to parameter updates. A second forward propagation is used instead of zeroing the loss of unselected samples after the first forward propagation because, in dynamic graph mechanisms like PyTorch, the first forward propagation has already established a complete computational graph for the entire batch. Subsequent zeroing typically only changes the gradient values ​​without substantially reducing the traversal and computational overhead of operators and tensors during the backpropagation phase, making it difficult to achieve real training acceleration. Performing a separate second forward pass to construct a sub-computation graph can reduce the tensor size and gradient path involved in backpropagation from the source, significantly reducing backpropagation overhead while introducing an additional forward pass, thereby achieving effective training acceleration.

[0057] This invention introduces a high-loss sample screening mechanism grouped by category. While prioritizing the selection of high-loss samples, it applies category quota / proportion constraints, avoiding the problems of category concentration and neglect of minority classes caused by screening only by loss. This allows the sample set in the backpropagation stage to achieve a balance between "information content (difficulty)" and "category coverage (balance)", improving the diversity and representativeness of gradient information, thereby enhancing the convergence stability of the training process and improving the final convergence quality.

[0058] This invention further adopts an update method of "secondary forward propagation + backpropagation only for selected samples", which can avoid the problem in the dynamic graph framework that "setting the loss of unselected samples to zero cannot substantially reduce the computational cost of backpropagation". This reduces the backpropagation computation graph from the source to only cover the operators and tensors corresponding to the backpropagation dataset, significantly reducing the computational cost and memory read / write overhead of backpropagation. It achieves verifiable real acceleration effects from the engineering implementation level, while improving the efficiency of computing resource utilization while maintaining or improving model performance.

[0059] The screening strategy of this invention can flexibly adjust the screening ratio and category quota at the batch level, making the method adaptable to the needs of different category distributions, different task forms and different training stages. It can also be integrated with existing optimizers, learning rate strategies and distributed training processes to achieve low-intrusion deployment. It is less likely to introduce class bias and training oscillations while reducing the number of backpropagation samples, and has better stability and controllability.

[0060] The effectiveness of the proposed method has been validated on a wide range of datasets, and experimental results demonstrate its superior performance in accelerating deep neural network training.

[0061] Next, refer to Figure 4 The method of the present invention will be further described below.

[0062] S101: Perform the first forward propagation on all samples in the current training batch, calculate the loss value for each sample, and obtain its corresponding class label.

[0063] In a training iteration, the current batch of data is input. This includes sample input x i Supervisory signal y i and the category label c used for category constraints i For all samples in this batch, in model f θ Perform a complete forward propagation, according to formula l i =l(f θ (x i ),y i The loss value l for each sample is calculated. i This forward propagation is only used to evaluate the importance of samples (i.e., the level of loss value) and obtain the grouping criteria (class labels). It does not trigger backpropagation or parameter updates, so the computational cost is much lower than that of the complete training steps.

[0064] S102: Based on the loss value and category label, group the samples by category within the batch, and within each category, select high-loss samples according to the loss value and a preset category retention ratio or quota, and merge them to form a backpropagation dataset.

[0065] Obtain the loss set of all samples within the batch. With category label set Next, high-loss sample screening by category is performed. First, according to the formula... Group the sample index by category k to obtain a subset corresponding to each category. Subsequently, a preset retention ratio ρ is applied to each category k. k ∈(0,1” (or directly specify quota q) k ), calculate the number of samples that should be included in this category. Next, in each category subset Internally, with loss value l i Sort the data in descending order using priority metrics, and select the top I with the largest loss values. k Each sample index constitutes the set of selected indices for that category. S = ∪ k I k Finally, the union of the selected index sets for all categories is taken to obtain the global selected sample index set S = ∪ k I k Based on this, the dataset actually used for backpropagation in this iteration was constructed. This step ensures that the selected sample set is biased towards samples that are more "difficult" (high loss) to the current model, while maintaining the balance and coverage between categories through category quota constraints.

[0066] S103: Perform a second forward and backward propagation only on samples in the backpropagation dataset to calculate gradients and update model parameters; for samples not selected in the backpropagation dataset, skip their backpropagation process.

[0067] Using only the reduced dataset constructed in step two Perform the second forward propagation. This forward propagation is... A separate, lightweight computational graph is reconstructed from the samples. Based on this computational graph, backpropagation is performed according to the formula... The gradient g is calculated only based on the selected samples, and then the optimizer performs parameter updates θ←Opt(θ, g). The key point is that for samples not selected in the original batch... The loss value calculated in step one is only used for evaluation. In this step, the corresponding computation graph node is completely excluded and does not participate in any gradient calculation or parameter update. This "secondary forward subgraph construction" method, compared to the pseudo-optimization of setting the loss of unselected samples to zero in the first complete computation graph, can substantially reduce the number of tensor operations and computational cost in the backpropagation process from the source, thereby achieving real training acceleration.

[0068] The method of this invention will be explained in detail below using a specific image classification training scenario as an example.

[0069] Hypothetical scenario:

[0070] A convolutional neural network (CNN) is used for training on the CIFAR-10 dataset (10 classes). The current training batch size is N = 20 samples. The preset sample retention ratio ρ for each class is... k=0.5 (meaning that half of the high-loss samples in each class are retained for backpropagation). In the current batch, the class distribution of the samples is: cat (3), dog (4), bird (2), car (4), deer (1), frog (2), horse (2), boat (1), airplane (1). The current model parameter is θ.

[0071] Step 1: Perform the first forward propagation on all samples in the current training batch, calculate the loss value of each sample, and obtain its corresponding class label.

[0072] Input batch data: 20 samples were loaded, each sample being a triple (x_i, y_i, c_i). Where x_i is a 32x32 RGB image tensor, y_i is its true class label (e.g., a one-hot vector), and c_i is the class index in integer form (0 to 9).

[0073] Perform forward propagation: Take these 20 images {x1, x2, ..., x...} 20 The data is input into the current CNN model f_θ all at once.

[0074] Calculate the loss per sample: The model outputs the predicted probability distribution for each sample. Using the cross-entropy loss function l(·), calculate the loss value l_i for each sample. Assume that after calculation, 20 loss values ​​are obtained, for example:

[0075] Sample 1 (cat): l1 = 2.3

[0076] Sample 2 (dog): l2 = 0.1

[0077] Sample 3 (bird): l3 = 1.8

[0078] ...and so on, until l 20 .

[0079] Result: After this step, two key lists were obtained:

[0080] Loss list: [l1=2.3,l2=0.1,l3=1.8,...,l 20 ]

[0081] Category label list: [c1,c2,c3,...,c 20 (For example, c1 = 3 (cats), c2 = 5 (dogs), etc.)

[0082] This step only performed forward computation and did not perform backpropagation, therefore the model parameters were not updated.

[0083] Step 2: Based on the loss value and class label, group the samples by class within the batch, and within each class, select high-loss samples according to the loss value and a preset retention ratio or quota, and merge them to form a backpropagation dataset.

[0084] Grouping by category: Iterate through the indices (1 to 20) of the 20 samples and assign them to 10 buckets based on their category label c_i. For example:

[0085] B_cat = {index of sample 1, index of sample 8, index of sample 15} (because there are 3 cats in the batch)

[0086] B_dog = {index of sample 2, index of sample 7, index of sample 12, index of sample 18} (4 dogs)

[0087] The same applies to other categories.

[0088] Calculate the number of entries in each category: based on the preset ratio ρ k =0.5, calculate the number of samples q_k that should be selected for each category (rounded up).

[0089] For cats: |B_cat| = 3,

[0090] For dogs: |B_dog| = 4,

[0091] For birds: |B_bird|=2,

[0092] For categories with only one sample (such as deer, boat, airplane): (That is, the sample will definitely be selected).

[0093] High-loss sorting and filtering within each category: Within the sample index set of each category, sort the samples in descending order according to the loss value l_i calculated in step one, and select the top q_k samples with the largest loss.

[0094] Taking cats as an example: Assume B_cat = {index 1, index 8, index 15}, and its corresponding loss is {l1 = 2.3, l8 = 0.5, l...} 15 =1.2}. After sorting by loss, the results are: index1 (2.3) > index15 (1.2) > index8 (0.5). Since q_cat = 2, the first two are selected, i.e., I_cat = {index1, index15}.

[0095] Taking dogs as an example: B_dog = {index 2, index 7, index 12, index 18}, the loss is {l2 = 0.1, l7 = 2.1, l 12 =0.3,l 18=1.9}. After sorting: index7(2.1)>index18(1.9)>index12(0.3)>index2(0.1). q_dog=2, therefore I_dog={index7, index18}.

[0096] Repeat this process for all categories.

[0097] Merge and construct the backpropagation dataset: Merge the selected index sets I_k for all categories: S = I_cat ∪ I_dog ∪ ... ∪ I_airplane. Assume the final S = {index 1, index 7, index 9, index 10, index 11, index 13, index 14, index 15, index 16, index 18, index 19, index 20}, a total of 12 indices (because some categories have only one sample and are selected). The sample selection rate of the original batch is 12 / 20 = 60%. Based on the index set S, extract the corresponding data from the original 20 samples to form a new, smaller batch B_S for subsequent key calculations.

[0098] Step 3: Perform a second forward and backward propagation only on samples in the backpropagation dataset to calculate gradients and update model parameters; skip the backpropagation process for samples not selected for the backpropagation dataset.

[0099] Data extraction: Extract the input x_i and label y_i of the 12 samples contained in B_S from memory.

[0100] Perform a second forward propagation: Input only these 12 samples {x_i|i∈S} into the same CNN model f_θ. Note that the model parameters θ have not changed since step one. This forward propagation will build a new, independent computational graph for these 12 samples.

[0101] Calculate the aggregate loss and gradient: On the new computation graph of these 12 samples, calculate their total loss L_S = Σ_{i∈S}l(f_θ(x_i),y_i). Then, perform backpropagation (backward()) only on the computation graph of these 12 samples to calculate the gradient. This gradient only reflects the impact of the 12 selected "difficult" and "class-balanced" samples on the model parameters.

[0102] Updating model parameters: The optimizer (such as SGD or Adam) uses the calculated gradient g to update the model parameters: θ←Opt(θ,g).

[0103] Key skip operation: The 8 samples that were not selected in the original batch (indices not in S, such as indices 2, 8, etc.) are not used at all in step three. The loss values ​​calculated for them in step one (e.g., l2 = 0.1, l8 = 0.5) are only used for the selection decision and are then discarded. They do not participate in the second forward propagation, their data do not form a new computational graph, and therefore they will not participate in the backpropagation and parameter updates of this iteration. This saves the overhead of calculating gradients for these 8 "simple" samples.

[0104] This iteration is now complete. Through the above process, only 12 out of 20 samples underwent full backpropagation in this training round, theoretically saving approximately 40% of the core computational (backpropagation) overhead. At the same time, due to the selection mechanism ensuring class coverage and prioritizing high-loss samples, the learning efficiency and stability of the model are guaranteed.

[0105] like Figure 5 As shown, in one specific embodiment, the present invention, based on the loss value and category label, groups samples by category within a batch, and within each category, selects high-loss samples according to the loss value and a preset category retention ratio or quota. Specifically, this includes:

[0106] S201: Divide the sample index within a batch into multiple category subsets based on the category labels;

[0107] S202: For each category subset, calculate the number of samples to be selected based on the preset retention ratio of that category, or directly use the preset quota;

[0108] S203: Within each category subset, select the top k sample indices based on the loss values ​​in descending order, where k is the calculated number of samples or a preset quota;

[0109] S204: Merge the sample indices selected from all category subsets to obtain the global selected sample set, and construct the backpropagation dataset accordingly.

[0110] For example, firstly, the sample index within a batch is divided into multiple category subsets based on the category labels.

[0111] Specifically, after obtaining all loss values ​​and class labels for the batch (N=20), create a dictionary or list array where the keys (or indices) are the class numbers (0 to 9). Iterate through the indices of the 20 samples (i from 1 to 20):

[0112] Read the category c_i of sample i (e.g., c1 = 3, representing the 'cat' class).

[0113] Add index i to the set corresponding to category c_i.

[0114] This forms a data structure such as class_to_indices = {3:[1,8,15],5:[2,7,12,18],2:[3,11],...}. Each list is a subset of a category B.

[0115] Secondly, for each category subset, the number of samples to be selected is calculated based on the preset retention ratio of that category.

[0116] Specifically, for each entry in class_to_indices (category k and its index list B) k ):

[0117] Get the length of the list n_k = len(B k ).

[0118] Query the preset retention ratio ρ for this type k (In this example, the global value is set to 0.5).

[0119] Calculate the number of candidates: qk = ceil(ρ k *n_k).

[0120] For example,

[0121] For B3 (cats, 3 samples): q3 = ceil(0.5 * 3) = 2

[0122] For B5 (dogs, 4 samples): q5 = ceil(0.5 * 4) = 2

[0123] For B2 (birds, 2 samples): q2 = ceil(0.5 * 2) = 1

[0124] For B9 (ship, 1 sample): q9 = ceil(0.5 * 1) = 1

[0125] Furthermore, within each category subset, a specified number of high-loss sample indices are selected based on the loss value.

[0126] Specifically, for each category k, execute:

[0127] Retrieve the loss value corresponding to the index in B from the global loss list.

[0128] Sort the indexes in descending order of their corresponding loss values.

[0129] Take the first qk indices of the sorted list.

[0130] For example:

[0131] The index of B3 (cat) is [1, 8, 15], with a loss of [2.3, 0.5, 1.2]. After sorting, the index is [1, 15, 8]. Taking the first q3 = 2 elements, we get I3 = [1, 15].

[0132] The index of B5 (dog) is [2,7,12,18], with a loss of [0.1,2.1,0.3,1.9]. After sorting, the index is [7,18,12,2]. Taking the first q5 = 2 elements, we get I5 = [7,18].

[0133] Other categories follow the same procedure.

[0134] Finally, the sample indices selected from all category subsets are merged to form the backpropagation dataset.

[0135] Specifically, create an empty list S. Iterate through the selection index set Ik for all categories and add all its indices to S.

[0136] For example, S = I3∪I5∪I2∪... = [1,15,7,18,3,...] (a total of 12 indices).

[0137] Finally, based on the index list S, the corresponding input data {x_i|iin S} and labels {y_i|iin S} are extracted from the tensors of the original batch using torch.index_select or a similar operation, forming a new tensor B_S with a batch size of 12.

[0138] In one embodiment, the category retention ratio or quota is a pre-set fixed value or a variable value that is dynamically adjusted during training based on the training stage, model state, or data distribution.

[0139] For example, in the early stages of training (e.g., the first 10 epochs), it is desirable for the model to have broad exposure to all samples, so a high global retention ratio is set, such as ρ = 0.8. In the middle stages of training (epochs 11 to 30), to accelerate and focus on difficult samples, ρ is reduced to 0.5. In the later stages of training (after the 30th epoch), to fine-tune the model and prevent overfitting to noise, ρ is slightly increased to 0.6.

[0140] For example, in step two of each iteration, the current training epoch and iteration number can be checked, and the appropriate ρ value can be found and applied according to the preset schedule.

[0141] In another embodiment, for balanced datasets like CIFAR-10, the proportions can be uniform. However, for long-tail datasets (such as ImageNet-LT), it is desirable to protect minority classes.

[0142] Specifically, a base ratio ρ_base = 0.4 is preset. For each class k, its retention ratio ρ in this batch is adjusted according to its frequency freq_k in the entire training set (calculated before training begins). k For example, using inverse frequency weighting: ρ k =ρ_base*(median_freq / freq_k)^0.5. In this way, classes with few samples will get a higher retention rate (possibly greater than ρ_base), and even reach 1.0 in some batches (i.e., all samples of that class are selected), to prevent them from being ignored.

[0143] In one embodiment of the invention, a separate computation graph is constructed for the backpropagation dataset by performing a second forward propagation, so that backpropagation is performed only on the reduced computation graph, thereby reducing the tensor size and computational cost involved in backpropagation.

[0144] Specifically, outputs = model(B_inputs) and losses = criterion(outputs, B_labels) are executed on the original 20 samples B. At this point, in the PyTorch dynamic graph, a computation graph containing the complete forward path of the 20 samples has been implicitly constructed, but backpropagation has not yet been performed.

[0145] In step three, instead of directly using the calculation graph from step one, we start over.

[0146] Specifically, using the index S obtained in step two, a new, independent tensor B_S is obtained through slicing operations of selected_inputs = B_inputs[S,:,:,:] and selected_labels = B_labels[S].

[0147] Execute `selected_outputs = model(selected_inputs)` and `final_loss = criterion(selected_outputs, selected_labels)`. This creates a completely new computational graph containing only the forward paths of 12 samples.

[0148] Call final_loss.backward(). At this point, the backpropagation engine will only traverse this new, smaller computation graph. The number of intermediate activation tensors and gradient tensors involved in the computation graph is reduced to approximately 12 / 20 = 60% of the original, thus substantially reducing the computational cost and memory access overhead of backpropagation.

[0149] In addition, such as Figure 6As shown, the present invention also provides a deep neural network training acceleration system 600 based on a sample priority mechanism, comprising:

[0150] Evaluation module 601 is used to perform the first forward propagation on the current training batch to calculate the loss value for each sample and obtain its class label;

[0151] The screening module 602, connected to the evaluation module, is used to receive the loss value and the category label, and perform high loss sample screening by category grouping to construct the backpropagation dataset;

[0152] The training update module 603, connected to the filtering module, is used to perform a second forward and backward propagation only on the backpropagation dataset to update the model parameters and skip the backpropagation of non-selected samples.

[0153] In addition, such as Figure 7 As shown, the filtering module 602 further includes:

[0154] Grouping unit 6021 is used to divide batch samples into multiple category subsets based on category labels;

[0155] Quota determination unit 6022 is used to determine the number of samples to be selected for each category subset;

[0156] Selection unit 6023 is used to select a specified number of high-loss sample indices within each category subset, sorted by loss value.

[0157] Merging unit 6024 is used to merge the sample indices selected from all category subsets to form the backpropagation dataset.

[0158] The functions and implementation methods of the various modules included in the deep neural network training acceleration system 600 based on the sample priority mechanism and the screening module 602 of the present invention have been referred to above. Figures 1-5 The description has already been provided, so I will not repeat it here.

[0159] In addition, the present invention also provides an electronic device comprising:

[0160] At least one processor;

[0161] And, a memory communicatively connected to the at least one processor;

[0162] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the above-mentioned reference. Figures 1-5 The method described.

[0163] Additionally, one or more embodiments of this disclosure can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a 'non-transitory computer-readable storage medium') to perform the functions of one or more embodiments of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more embodiments of the above embodiments, and by a method performed by a computer of the system or device, for example by reading and executing computer-executable instructions from the storage medium to perform the functions of one or more embodiments of the above embodiments and / or controlling one or more circuits to perform the functions of one or more embodiments of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a separate computer or a network of separate processors for reading and executing computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash memory devices, memory cards, etc.

[0164] The embodiments of this disclosure can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads and executes the program.

[0165] Although this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be interpreted in the broadest possible sense to cover all such modifications and equivalent structures and functions.

Claims

1. A method for accelerating deep neural network training based on a sample priority mechanism, characterized in that, Includes the following steps: Step 1: Perform the first forward propagation on all samples in the current training batch, calculate the loss value for each sample, and obtain its corresponding class label; Step 2: Based on the loss value and category label, group the samples by category within the batch, and within each category, select high-loss samples according to the loss value and a preset category retention ratio or quota, and merge them to form a backpropagation dataset; Step 3: Perform a second forward and backward propagation only on samples in the backpropagation dataset to calculate gradients and update model parameters; for samples not selected in the backpropagation dataset, skip their backpropagation process.

2. The method according to claim 1, characterized in that, The step of grouping samples by category within a batch based on the loss value and category label, and then selecting high-loss samples within each category according to a preset retention ratio or quota based on the loss value, specifically includes: Divide the sample index within a batch into multiple category subsets based on the category labels; For each category subset, calculate the number of samples to be selected based on the preset retention ratio for that category, or directly use the preset quota; Within each category subset, the top k sample indices are selected in descending order of loss values, where k is the calculated number of samples or a preset quota; The sample indices selected from all category subsets are merged to obtain the global selected sample set, and the backpropagation dataset is constructed accordingly.

3. The method according to claim 1 or 2, characterized in that, The category retention ratio or quota is a fixed value set in advance, or a variable value that is dynamically adjusted during the training process according to the training stage, model status, or data distribution.

4. The method according to claim 1, characterized in that, In step three, a separate computation graph is constructed for the backpropagation dataset by performing a second forward propagation, so that backpropagation is performed only on the reduced computation graph, thereby reducing the tensor size and computational load involved in backpropagation.

5. The method according to claim 1, characterized in that, The method is applicable to deep learning frameworks that employ dynamic graph mechanisms. By constructing a sub-computation graph through the second forward propagation, it avoids the problem that the operation of zeroing the loss of unselected samples cannot substantially reduce computational overhead.

6. The method according to claim 1, characterized in that, The method is integrated into the training iteration of standard stochastic gradient descent or its variants, and is compatible with different optimizers, learning rate adjustment strategies, and distributed training frameworks.

7. A deep neural network training acceleration system based on a sample priority mechanism, characterized in that, include: The evaluation module performs the first forward propagation on the current training batch to calculate the loss value for each sample and obtain its class label; A filtering module, connected to the evaluation module, is used to receive the loss value and class label, and perform high-loss sample filtering by class grouping to construct a backpropagation dataset; The training update module, connected to the filtering module, is used to perform a second forward and backward propagation only on the backpropagation dataset to update the model parameters and skip the backpropagation of non-selected samples.

8. The system according to claim 7, characterized in that, The filtering module further includes: Grouping units are used to divide batch samples into multiple category subsets based on category labels; Quota determination unit, used to determine the number of samples to be selected for each category subset; The selection unit is used to select a specified number of high-loss sample indices within each category subset, sorted by loss value. The merging unit is used to merge the sample indices selected from all category subsets to form the backpropagation dataset.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images