An online batch data processing method and device for training a deep neural network

Abstract

The present application relates to an online batch data processing method and device for training deep neural networks, the method comprising the following steps: sampling a data batch from a given dataset; selecting a subset S from the data batch by using an orthogonalization-based objective function; training a target deep neural network based on the subset S. Compared with the prior art, the present application uses an orthogonalization method to remove the redundancy between subsets, provides an objective function based on orthogonalization de-redundancy correlation to measure the diversity and representativeness of the subsets at the same time, and provides an efficient optimization method based on the objective function and a corresponding data selection algorithm, which can efficiently select diverse and representative subsets.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an online batch data processing method and apparatus for training deep neural networks. Background Technology

[0002] In recent years, deep learning models have made tremendous progress in various fields such as computer vision and natural language understanding. These advances are based on increasingly larger model sizes and exponentially growing training data volumes, but on the other hand, they have also led to enormous computational and time costs for training modern deep learning models.

[0003] Existing batch data selection methods can be broadly categorized into two types: those that rely on an additional reference model and those that do not. The paper "Prioritized training on points that are learnable, worth learning, and not yet learnt" discloses a method for selecting valuable samples using a reference model trained with additional data. However, this method depends on an additional reference model, limiting its practical application when such an ideal model is unavailable. Furthermore, these methods are confined to classification tasks and cannot be extended to a wider range of tasks.

[0004] In summary, the main drawback of existing batch data selection methods designed for deep neural networks is their inability to effectively guarantee the diversity of the selected sample subset, leading to the selection of redundant samples and affecting model convergence. This is because these methods independently score and select samples, failing to consider correlations and redundancy between samples. Furthermore, some methods rely on additional reference models and are limited to accelerating classification tasks, thus restricting their application scope. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing an online batch data processing method and device for training deep neural networks. By using an orthogonalized objective function to select a diverse and representative subset from the data batch for training, the invention solves or partially solves the problem of high training sample redundancy and poor representativeness leading to unsatisfactory training results.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] One aspect of the present invention provides an online batch data processing method for training deep neural networks, comprising the following steps:

[0008] A data batch is obtained by sampling from a given dataset;

[0009] A subset S is selected from the data batch using an orthogonalization-based objective function;

[0010] The target deep neural network is trained based on a subset S.

[0011] The process of selecting subset S includes the following steps:

[0012] Initialize the orthogonal basis E and subset S to an empty set, and initialize the feature synthesis Sum to the sum of all features in the data batch;

[0013] Based on the gradient of the parameters of the last layer of the target deep neural network, and based on the gradient and the current feature synthesis Sum, a sample is extracted from the data batch, the contribution e of the sample to the current orthogonal basis is calculated and the orthogonal basis E is updated, the extracted sample is added to the subset S, the orthogonal component of the sample is deleted from the data batch, and this step is repeated until the preset termination condition is met.

[0014] As a preferred technical solution, the extraction of a sample from the data batch is achieved using the following formula:

[0015]

[0016] Where d is the extracted sample, B is the current data batch, θ is the parameter of the target deep neural network, and g(d,θ) is the gradient of the parameters of the last layer of the target deep neural network.

[0017] As a preferred technical solution, the initialization of the feature synthesis Sum to the sum of all features in the data batch is achieved by the following formula:

[0018]

[0019] Where B is the current data batch, θ is the parameter of the target deep neural network, g(B,θ) is the gradient of the last layer of the network corresponding to each data point in B, ← indicates assignment, and u represents an element of g(B,θ), that is, the gradient of the last layer of the network corresponding to a data point in B.

[0020] As a preferred technical solution, the contribution e of the sample to the current orthogonal basis is calculated using the following formula:

[0021]

[0022] Where ← denotes assignment, g(d,θ) is the gradient, E=ε(g(S,θ)), and ε() represents the set of all orthonormal bases in a subspace composed of multiple elements.

[0023] As a preferred technical solution, the removal of orthogonal components of samples from the data batch is achieved using the following formula:

[0024] Sum←Sum-(e·Sum)e

[0025] Here, ← indicates assignment.

[0026] As a preferred technical solution, the termination condition is: feature synthesis Sum = 0 or |S| = N. S , where N S This is the default value.

[0027] As a preferred technical solution, the objective function is:

[0028]

[0029]

[0030] Where d represents the extracted samples, B represents the current data batch, θ represents the parameters of the target deep neural network, and N... S Here are the preset parameters, g(B,θ) is the gradient of the parameters of the last layer of the target deep neural network, u represents an element of g(B,θ), and ε() represents the set of all orthonormal bases in the subspace composed of multiple elements.

[0031] In another aspect, the present invention provides an image data processing method, which performs image recognition based on an acquired image to be processed using a pre-trained deep neural network to obtain an image recognition result, wherein the deep neural network is trained using the aforementioned online batch data processing method for training deep neural networks.

[0032] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned online batch data processing method for training a deep neural network.

[0033] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, said one or more programs including instructions for performing the aforementioned online batch data processing method for training a deep neural network.

[0034] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0035] (1) The selected sample subset has strong diversity and good training effect: This invention uses an orthogonalized objective function to select a representative subset S from the data batch and trains the target deep neural network based on the subset S. By reducing the redundancy of the samples in the subset, the representativeness and diversity of the samples are improved. The deep neural network can achieve higher performance with fewer training rounds. The deeper the network and the larger the training dataset, the more obvious the acceleration effect of this method.

[0036] (2) Strong applicability: By selecting the gradient of the last layer of the network as the basis for data selection, this method can be applied in principle to any current deep neural network training acceleration based on backpropagation and gradient update.

[0037] (3) Wide range of applications: This method can be used in a wide range of tasks, including recognition, segmentation, and retrieval.

[0038] (4) Achieve data rebalancing and debiasing: This method can achieve additional rebalancing and debiasing effects on some imbalanced or biased datasets. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the online batch data processing method for training a deep neural network in this embodiment. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] Example 1

[0042] To address the problems existing in the prior art, this embodiment provides an online batch data processing method for training deep neural networks, which achieves efficient batch selection of diverse and representative samples without the need for a reference model, thereby accelerating the training process while maintaining model performance.

[0043] See Figure 1 Consider training a deep neural network f on training data D. θ The parameter is θ, and the method includes the following steps:

[0044] S1, a batch of data is obtained by sampling from a given dataset.

[0045] In each training step, a data batch B is sampled from the training set D to update the model, containing N... B There are 10 data points. Specifically, data batch B here is obtained through random sampling.

[0046] S2, a subset S is selected from the data batch using an orthogonalized objective function.

[0047] Online batch data selection involves sampling a subset S from B to update the model, thereby reducing training overhead. The number of subsets S is limited to N. s This embodiment considers both the diversity and representativeness of subset S, providing an index to measure the orthogonalization and redundancy-removal relevance of the subset as a whole relative to the entire set, thus balancing the diversity and representativeness of the subset:

[0048]

[0049] In the formula, g represents the function for extracting data from the original data and selecting the required features. In this embodiment, the gradient of the last layer parameters of the neural network is used as the feature for selection. g(S,θ) is the gradient corresponding to each element in the set, and g(d,θ) is the gradient corresponding to the d-th data point.

[0050] It should be noted that direct network output, intermediate layer features, and full network parameters can also be used as alternative features. ε represents the set of all possible orthonormal bases for a given normal subspace. Therefore, the objective function for selecting the subset S is:

[0051]

[0052] Furthermore, this embodiment provides another computational form of the r function that holds for any orthonormal basis, thus simplifying the complex optimization process for orthonormal bases:

[0053]

[0054] For the objective function in the above equation, this embodiment provides a fast data selection algorithm that can obtain diverse and representative subsets S that meet the requirements. It should be noted that the following algorithm can be applied to different selection paradigms, such as selecting subsets from the overall training set. The specific steps of the algorithm are:

[0055] Step 1: Initialize the selected subset S as an empty set, and initialize the corresponding orthogonal basis E as an empty set. Initialize Sum as the sum of all features of the elements in the entire data batch B: Sum←∑ u∈g(B,θ) u.

[0056] Step 2, add a sample to S that approximately maximizes the current r. This step includes the following sub-steps:

[0057] Step 201, in the current step, select the sample d to be added: d ← argmax d∈B |g(d,θ)·Sum|.

[0058] Step 202, calculate the contribution of sample d to the current orthogonal basis:

[0059] Step 203, update E, S, B: S←S∪{d}, E←E∪{e}, B←B\{d}.

[0060] Step 204, subtract the orthogonal component of the selected sample from Sum: Sum←Sum-(e·Sum)e.

[0061] Step 3, repeat Step 2 until |S| = N S Or Sum = 0, which is the termination condition.

[0062] It should be noted that the dot product correlation (corresponding to the "·" in the formula) can be appropriately replaced, such as cosine correlation, Euclidean distance, etc.

[0063] This method only uses the gradient of the last layer as the basis for data selection; therefore, it can, in principle, be applied to accelerating the training of any deep neural network currently based on backpropagation and gradient updates. The deeper the network and the larger the training dataset, the more significant the acceleration effect of the method.

[0064] In practice, this method can be used in a wide range of tasks, including recognition, segmentation, and retrieval. Furthermore, because the algorithm emphasizes data diversity, it can additionally achieve rebalancing and debiasing effects on imbalanced or biased datasets.

[0065] S3 trains the target deep neural network based on a subset S.

[0066] To verify the effectiveness of this method, performance was validated on multiple benchmark datasets, including seven datasets for image recognition, imbalanced classification, semantic segmentation, and multimodal retrieval. All experiments demonstrated the effectiveness of this method in accelerating deep network training while maintaining model performance. Specifically, with a 70% reduction in training iterations, the accuracy on image recognition tasks decreased by an average of less than 0.5% (absolute value), the performance on segmentation tasks decreased by less than 1% (absolute value), and the cross-modal retrieval performance even improved compared to full training, demonstrating a significant advantage over other similar methods.

[0067] This method utilizes orthogonalization to remove redundancy among subsets, provides an objective function based on orthogonalization to simultaneously measure the diversity and representativeness of subsets, and offers an efficient optimization method and corresponding data selection algorithm based on the objective function, which can efficiently select diverse and representative subsets.

[0068] This method has a wide range of applications and can be used to accelerate the training of many deep learning tasks such as classification, segmentation, and retrieval. It does not rely on additional reference models and can be applied at low cost. It can effectively enhance the diversity of data selection and has achieved excellent acceleration performance on multiple benchmark datasets for various tasks.

[0069] Example 2

[0070] This embodiment provides an image data processing method. Based on the acquired image to be processed, a pre-trained deep neural network is used for image recognition to obtain the image recognition result. The deep neural network is trained using the online batch data processing method for training deep neural networks as described in Embodiment 1.

[0071] It should be noted that the specific structure of deep neural networks is not limited here. Existing structures can be used as long as there is no conflict.

[0072] It should be noted that, without conflict, deep neural networks can also be used to perform other tasks such as image segmentation and retrieval.

[0073] Example 3

[0074] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the online batch data processing method for training a deep neural network as described in Embodiment 1.

[0075] Example 4

[0076] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing an online batch data processing method for training a deep neural network as described in Embodiment 1.

[0077] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An online batch data processing method for training deep neural networks, characterized in that, Based on the acquired image to be processed, image recognition is performed using a pre-trained deep neural network to obtain the image recognition result. The deep neural network is trained using an online batch data processing method, which includes the following steps: A data batch is obtained by sampling from a given dataset; Selecting a subset from the data batch using an orthogonalized objective function ; Based on subsets Train the target deep neural network. Among them, selecting subsets The process includes the following steps: orthogonal base and subsets Initialize to an empty set and synthesize features. Initialize to the sum of all features in the data batch; Based on the gradient of the parameters of the last layer of the target deep neural network, and based on the gradient and the current feature synthesis... A sample is drawn from the data batch, and the contribution of the sample to the current orthogonal basis is calculated. And update the orthogonal basis Add the extracted samples to the subset Remove the orthogonal components of the samples from the data batch and repeat this step until the preset termination condition is met.

2. The online batch data processing method for training deep neural networks according to claim 1, characterized in that, The extraction of a sample from the data batch is achieved using the following formula: in, For the sample drawn, For the current data batch, For the parameters of the target deep neural network, This represents the gradient of the parameters of the last layer of the target deep neural network.

3. The online batch data processing method for training deep neural networks according to claim 1, characterized in that, The aforementioned feature synthesis The initialization to the sum of all features in the data batch is achieved using the following formula: in, For the current data batch, For the parameters of the target deep neural network, for The gradients of the last layer of the network corresponding to each of the data points are as follows: This indicates assignment. express One of the elements, namely The gradient of the last layer of the network corresponding to a data point.

4. The online batch data processing method for training deep neural networks according to claim 1, characterized in that, The contribution of the sample to the current orthogonal basis Calculate using the following formula: in, This indicates assignment. For gradient, , It represents the set of all orthonormal bases in a subspace consisting of multiple elements.

5. The online batch data processing method for training deep neural networks according to claim 1, characterized in that, The removal of orthogonal components of samples from the data batch is achieved using the following formula: in, This indicates assignment.

6. The online batch data processing method for training deep neural networks according to claim 1, characterized in that, The termination condition is: feature synthesis or ,in This is the default value.

7. The online batch data processing method for training deep neural networks according to claim 1, characterized in that, The objective function is: in, For the sample drawn, For the current data batch, For the parameters of the target deep neural network, These are preset parameters. The gradient of the parameters of the last layer of the target deep neural network. express One of the elements, It represents the set of all orthonormal bases in a subspace consisting of multiple elements.

8. An electronic device, characterized in that, include: One or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for performing the online batch data processing method for training a deep neural network as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It includes one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the online batch data processing method for training a deep neural network as described in any one of claims 1-7.

Images