Method and device for training artificial neural network

Abstract

A method for training an artificial neural network includes: obtaining M training samples, each of the M training samples containing at least one target, where M is an integer greater than or equal to 2 (S510); dividing the M training samples into K groups of training samples based on the targets contained in the M training samples, the K groups of training samples corresponding one-to-one to K computing units, the K computing units being used to process the targets in the K groups of training samples, and the K computing units being used to train the artificial neural network based on the processing results of the targets in the K groups of training samples, where K is an integer greater than or equal to 2 (S520). This method ensures that the number of training samples contained in each group of training samples is equal or approximately equal, thereby reducing the difference in the time required for different computing units to process targets. The next step can only be performed in advance based on the processing results of all targets, thereby improving the training efficiency of the artificial neural network.

Description

Technical Field

[0001] The present invention relates to the field of artificial neural networks, and in particular to a method and apparatus for training an artificial neural network. Background Art

[0002] Artificial neural networks (ANNs) are the foundation of artificial intelligence. They require a large number of sample training to achieve specific functions, such as recognizing the content of an image or translating text.

[0003] To speed up ANN training, one method involves training the ANN through a distributed computing system. For example, synchronous stochastic gradient descent (SSGD) is used to train the ANN. The SSGD method slices the training sample to generate multiple slices of data. Each slice is then sent to a computing unit in the distributed computing system for forward propagation. After obtaining the loss function from the forward propagation, backpropagation is performed to determine the gradient of the loss function. Finally, the gradient values ​​obtained from each computing unit are aggregated, and the ANN is updated based on the aggregated gradient. These steps are repeated until training is complete.

[0004] The key to training ANNs using the SSGD method is to synchronize the processing rates of each computing unit, so that training can be completed at a faster training rate. For some ANNs that require multi-stage sample processing, such as the faster region-based convolutional neural network (RCNN) or Mask RCNN, the objects extracted from the training samples in the first stage are related to the sample content. This leads to an uneven load on the various computing units in the distributed computing system during the second stage of calculation. The computing units with heavier loads take longer to calculate the gradient values, causing the gradient aggregation process to be blocked by the computing units with heavier loads, ultimately reducing the ANN training efficiency. Summary of the Invention

[0005] The present application provides a method and apparatus for training ANN, which classifies training samples using a target-based classification method, thereby improving the efficiency of training ANN in a distributed training system.

[0006] In a first aspect, a method for training an ANN is provided, comprising: obtaining M training samples, each of the M training samples containing at least one target, where M is an integer greater than or equal to 2; dividing the M training samples into K groups of training samples according to the targets contained in the M training samples, the K groups of training samples corresponding one-to-one to K computing units, the K computing units being used to process the targets in the K groups of training samples, and the K computing units being used to train the ANN based on the processing results of the targets in the K groups of training samples, where K is an integer greater than or equal to 2.

[0007] The method for training ANN provided in the present application groups training samples based on the targets contained in the training samples. In the ANN training process where the targets in the training samples need to be processed synchronously, the number of training samples contained in each group of training samples can be determined according to the actual situation of the distributed computing system. For example, the number of training samples contained in each group of training samples is equal or approximately equal. In this way, the difference in the time required for different computing units to process targets can be reduced, and the next processing step that needs to be performed based on the processing results of all targets can be performed in advance, thereby improving the training efficiency of the ANN.

[0008] Optionally, dividing the M training samples into K groups of training samples according to the targets included in the M training samples includes: dividing the M training samples into K groups of training samples according to the targets included in the M training samples and the computing capabilities of the K computing units, wherein the number of targets included in the first group of training samples matches the computing capabilities of the first computing unit, the first group of training samples is any one of the M groups of training samples, and the first computing unit is a computing unit among the K computing units corresponding to the first group of training samples.

[0009] M training samples are grouped according to the computing power of each computing unit so that the number of targets processed by each computing unit matches the processing power of the computing unit. In this way, the difference in the time required for different computing units to process targets can be reduced, and the next step of processing that requires the processing results of all targets can be carried out in advance, thereby improving the training efficiency of the ANN.

[0010] Optionally, the M training samples are divided into K groups of training samples according to the targets contained in the M training samples, including: arranging the M training samples into a sample queue, wherein, for any two adjacent training samples in the sample queue, the number of targets contained in the previous training sample is less than or equal to the number of targets contained in the next training sample, or the number of targets contained in the previous training sample is greater than or equal to the number of targets contained in the next training sample; dividing the sample queue into M / (n·K) groups of training samples in sequence, where n is the number of training samples that can be processed at one time by any one of the K computing units, wherein the number of training samples that can be processed at one time by each of the K computing units is the same; and extracting M / n training samples from the M / (n·K) groups of training samples according to an extraction rule to obtain K groups of training samples, wherein any group of training samples in the K groups of training samples includes n training samples in any one of the M / (n·K) groups of training samples, wherein the extraction rule is: n training samples are extracted from each group of training samples in the M / (n·K) groups of training samples each time.

[0011] When the processing capabilities of each computing unit in a distributed training system are the same, the M training samples can be grouped according to the above scheme so that the number of targets contained in each group of training samples is the same or approximately the same. In this way, the time difference in generating gradient values ​​between the computing unit with the fastest computing rate and the computing unit with the slowest computing rate becomes smaller, allowing synchronous gradient aggregation to be carried out in advance, thereby improving the training efficiency of the ANN.

[0012] Optionally, n training samples are extracted from each of the M / (n·K) groups of training samples each time, including: n training samples are randomly extracted from each of the M / (n·K) groups of training samples each time.

[0013] The above scheme can further reduce the difference in the number of targets contained in each group of training samples in the K groups of training samples.

[0014] Optionally, the method further includes: shuffling each of the K groups of training samples; and sending the shuffled K groups of training samples to K processing units respectively.

[0015] The above scheme can disrupt the order of training samples in each group of training samples, enhance the randomness of the number of targets processed by each computing unit in a single batch, and avoid the problem of large differences in the number of targets when different computing units process a single batch of targets.

[0016] Optionally, dividing M training samples into K groups of training samples is a first grouping mode, the first grouping mode is one of at least two preset grouping modes, and the second grouping mode of the at least two grouping modes is: dividing the M training samples into K groups with the same or approximately the same number of training samples; dividing the M training samples into K groups of training samples according to the targets contained in the M training samples, including: obtaining indication information, the indication information is used to indicate the selection of the first grouping mode from at least two grouping modes; dividing the M training samples into K groups of training samples according to the indication information and the targets contained in the M training samples.

[0017] The above scheme provides users with more options, allowing them to flexibly choose a grouping method suitable for the current ANN training scenario according to actual conditions, thereby improving the ANN training efficiency.

[0018] In a second aspect, the present application provides an apparatus for training an ANN, which can implement the functions corresponding to the various steps in the method of the first aspect described above. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more units or modules corresponding to the above functions.

[0019] In one possible design, the apparatus includes a processor configured to support the apparatus in performing the corresponding functions of the method of the first aspect. The apparatus may also include a memory coupled to the processor and storing program instructions and data necessary for the apparatus. Optionally, the apparatus also includes a communication interface configured to support communication between the apparatus and other devices.

[0020] In a third aspect, the present application provides a computer program product, comprising: a computer program code, which, when executed by a processor of an ANN training device (e.g., a server), causes the ANN training device to perform the method of the first aspect.

[0021] In a fourth aspect, the present application provides a computer storage medium for storing computer software instructions used for the above-mentioned apparatus for training ANN, which includes a program designed for executing the method of the first aspect.

[0022] In a fifth aspect, the present application provides an ANN training system, comprising the apparatus of the second aspect, the computer program product of the third aspect, and the computing storage medium of the fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the architecture of a distributed computing system applicable to the technical solution of the present application;

[0024] Figure 2 Schematic diagram of the architecture of Mask RCNN applicable to this application;

[0025] Figure 3 This is a schematic flowchart of a training Mask RCNN provided by this application;

[0026] Figure 4 This is a schematic diagram of an ANN training interface provided by this application;

[0027] Figure 5 This is a schematic diagram of a method for dividing training samples during ANN training provided by the present application;

[0028] Figure 6 is a schematic diagram of a method for training ANN provided in this application;

[0029] Figure 7 Schematic diagram of an ANN training device provided by the present application;

[0030] Figure 8 is a schematic diagram of another apparatus for training ANN provided in the present application;

[0031] Figure 9 This is a schematic diagram of a system for training ANN provided in this application. DETAILED DESCRIPTION

[0032] In order to facilitate understanding of the technical solution of this application, a brief introduction to the concepts involved in this application is first given.

[0033] The work of each layer in ANN can be expressed mathematically From a physical perspective, the work of each layer in the ANN can be understood as completing the transformation from the input space to the output space (i.e., from the row space to the column space of the matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimensionality increase / decrease; 2. Zoom in / out; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the object being classified is not a single thing, but a class of things, and space refers to the collection of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in the neural network of this layer. The vector W determines the spatial transformation from the input space to the output space mentioned above, that is, the weight W of each layer controls how to transform the space. The purpose of training a deep neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of ANN is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0034] Because we want the output of the ANN to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the deep neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict lower, and continue to adjust until the neural network can predict the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value." This is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this output value as much as possible.

[0035] The loss function is usually a multivariable function, and the gradient can reflect the rate of change of the output value of the loss function when the variable changes. The larger the absolute value of the gradient value, the greater the rate of change of the output value of the loss function. The gradient value of the loss function when updating different parameters can be calculated, and the parameters can be continuously updated along the direction where the gradient value decreases fastest to reduce the output value of the loss function as soon as possible.

[0036] Below, the technical solution provided by this application will be described in detail with reference to the accompanying drawings.

[0037] Figure 1 The present invention is a schematic diagram of the architecture of a distributed computing system applicable to the technical solution of the present application.

[0038] The system 100 includes four computing units (workers) and a parameter server (PS). The four computing units (worker0, worker1, worker2, and worker3) perform forward propagation and backpropagation calculations on input training samples (e.g., images or sentences) to obtain gradient values ​​of the loss function (g0, g1, g2, and g3). Subsequently, the four computing units send their respective gradient values ​​to the PS, which synchronously aggregates the gradient values ​​to obtain the aggregated result g, which is the mean of the gradient values ​​of all computing units participating in distributed training. This result is used to update the ANN model parameters W′, obtaining the updated model parameters W. Finally, the PS distributes W to each computing unit, updating the model parameters of each computing unit, thus completing one iteration of training.

[0039] The computing unit may be a graphics processing unit (GPU), a tensor processing unit (TPU), or another type of computing unit, such as a central processing unit (CPU).

[0040] The PS can be located in the same device as the four computing units, or in different devices.

[0041] It should be understood that Figure 1 The system 100 shown is only an example of a technical solution applicable to the present application. The number of computing units and PSs included in the system 100 may also be other numbers. The system 100 may also include other units, such as an input unit.

[0042] The system 100 can train a variety of ANNs. Below, the method for training ANNs provided in this application is described using Mask RCNN as an example.

[0043] Figure 2 A schematic diagram of the architecture of a Mask RCNN suitable for this application is shown.

[0044] Mask RCNN is a convolutional neural network based on the improved Faster RCNN. It adds a branch network (i.e., Mask branch) on the basis of Faster RCNN, which enables target pixels to be segmented while achieving target detection. It can be used in combination with multiple RCNNs and has strong generalization ability.

[0045] The process of Mask RCNN processing pictures is as follows:

[0046] (a) Input the image into the backbone network (i.e., the base network of Faster RCNN) to extract features and obtain a feature map. The backbone network can be ResNet50, which stands for a 50-layer deep residual network (ResNet).

[0047] (b) Processing the feature map through a region proposal network (RPN), i.e., performing RPN forward propagation calculations to generate one or more proposal windows (proposals) for each feature map. For example, the RPN performs anchor decoding and non-maximum suppression (NMS) processing to generate one or more proposal windows (i.e., regions of interest).

[0048] (c) Different numbers of proposal windows are selected and fed into the region of interest (ROI) alignment layer. The ROI alignment layer normalizes the size of the proposal windows of different sizes. Simultaneously, the RPN calculates the RPN loss function and performs RPN backpropagation to obtain the gradient value of the RPN loss function. The RPN loss function is used to evaluate whether a region is a potential proposal window. Subsequently, the RPN aggregates the gradient values ​​of the RPN loss function and updates the RPN model parameters. This is done by performing gradient aggregation and model parameter updates on the RPN branches.

[0049] (d) The proposed window is processed by the Faster RCNN branch to obtain the classification loss function and regression loss function. Among them, the classification loss function is used to evaluate the type of object in the ROI, and the regression loss function is used to evaluate the deviation between the coordinates of the candidate box mapped to the original image and the coordinates of the correct annotation label (ground truth label).

[0050] (e) Synchronize the gradient values ​​of each computing unit to perform gradient aggregation and model parameter update operations for faster RCNN branches, as well as gradient aggregation and model parameter update operations for the backbone network.

[0051] (f) When the training results of the backbone network, RPN and Faster RCNN branches have converged, freeze the model parameters and then train the mask branch (the training of the mask branch can also be performed simultaneously with the training of the Faster RCNN branch). Figure 2The mask loss function shown is used to evaluate the deviation between the predicted result and the correct annotation label (ground truth label) of the segmented instance.

[0052] Figure 3 The flowchart of the system 100 for training the Mask RCNN is shown.

[0053] like Figure 3 As shown in the figure, the training sample of Mask RCNN is a dataset containing multiple images. Each computing unit has a list of the file names of the dataset and can independently and concurrently randomly extract several images from the dataset according to the file name list and add them to the queue (input pipeline).

[0054] Taking computing unit 0 as an example, computing unit 0 first pre-processes the pictures in queue 0, that is, it packages the pictures in the queue according to the number of pictures to be processed at a time to obtain a minibatch. Then, the minibatch is sent to the backbone network and executed. Figure 2 The steps shown.

[0055] From the process of Mask RCNN processing images, we can see that after step (c), since the faster RCNN branch processes the suggestion windows, the number of suggestion windows selected in each image is different, for example, Figure 3 The number of suggestion windows 0 and 1 shown is different. Therefore, the number of suggestion windows processed by the faster RCNN branch of each computing unit is different, which makes the time for the faster RCNN branches of different computing units to generate gradient values ​​during back propagation calculation different. Each computing unit needs to wait for the faster RCNN branch of the computing unit with the latest gradient value generation time to complete the back propagation calculation before it can perform synchronous gradient aggregation, which leads to a decrease in the training efficiency of Mask RCNN.

[0056] This application provides a method for training ANN, which is applied to Figure 1 The distributed computing system shown in FIG. 100 may include a user interface including options such as input data set, hyper parameter, and shuffle, as shown in FIG. Figure 4 shown.

[0057] The Dataset option allows users to select different training samples. Users can use this option to select appropriate training samples. For example, when training Mask RCNN, users can select Dataset 1, which can be the ImageNet dataset or the COCO dataset. When training a natural language processing network, users can select Dataset 2, which can be the Penn Treebank (PTB) dataset.

[0058] The hyperparameter option is used to provide users with the function of selecting different training models. Users can use this option to set the learning rate, number of iterations, number of ANN layers, and the number of neurons in each layer.

[0059] The shuffle option is used to provide users with the function of selecting different training sample grouping modes. Users can select different grouping modes according to actual conditions.

[0060] For example, if each computing unit does not need to process the targets in the training samples during the ANN training process, or if each computing unit can asynchronously process the targets in the training samples during the ANN training process, the user can select "normal global shuffle". After each computing unit in the system 100 obtains the "normal global shuffle" command, it randomly extracts an equal number of training samples from the data set and inputs them into its respective queues.

[0061] For another example, if each computing unit needs to synchronously process the targets in the training samples during the ANN training process, the user can select "target-based shuffle". After each computing unit in the system 100 receives the "target-based shuffle" command, it randomly extracts training samples with an equal number of targets from the data set and adds them to their respective queues.

[0062] The above method is only an example. The function of the shuffling option provided in this application is not limited to this. The shuffling option can also provide users with more shuffling functions, for example, shuffling based on target type so that the target types of the training samples input by each computing unit are the same.

[0063] After receiving the command of the data set option and the instruction information of "shuffle based on target", each computing unit in the system 100 can execute Figure 5 The method 500 is shown. The method 500 may be executed by the overall controller of the system 100 (eg, a CPU or a dedicated processor).

[0064] S510: Obtain M training samples, where each of the M training samples contains at least one target, and M is an integer greater than or equal to 2.

[0065] The training samples can be images or sentences. When the training samples are images, the targets can be objects in the images; when the training samples are sentences, the targets can be words or letters in the sentences.

[0066] S520, dividing the M training samples into K groups of training samples according to the targets contained in the M training samples, the K groups of training samples correspond one-to-one to K computing units, the K computing units are used to process the targets in the K groups of training samples, and the K computing units are used to train the ANN based on the processing results of the targets in the K groups of training samples, where K is an integer greater than or equal to 2.

[0067] When method 500 is applied to system 100, K is equal to 4. The value of M can be 20, that is, there are 20 training samples in the current data set, where each training sample contains at least one target. System 100 can determine the number of targets contained in the training sample based on the label of the training sample. For example, the number of targets contained in the 20 training samples is 40.

[0068] System 100 can divide the 20 training samples into 4 groups according to the labels of the training data, each group of training samples contains 10 targets (the number of training samples contained in each group of training samples may be different), and the 4 groups of training samples correspond to the 4 computing units of system 100 respectively. The 4 computing units are used to perform synchronous gradient aggregation processing on the targets in the 4 groups of training samples, and update the model parameters of the ANN according to the results of the synchronous gradient aggregation processing.

[0069] The number of targets included in each group of training samples mentioned above is only an example. The training samples can also be divided into 4 groups according to the following number.

[0070] Method 1: Group 1, 8 targets; Group 2, 12 targets; Group 3, 8 targets; Group 4, 12 targets.

[0071] Method 2: Group 1, 11 targets; Group 2, 9 targets; Group 3, 12 targets; Group 4, 8 targets.

[0072] Method 3: Group 1, 1 target; Group 2, 9 targets; Group 3, 1 target; Group 4, 9 targets.

[0073] In summary, the method 500 for training ANN provided in this application groups training samples based on the targets contained in the training samples. During the ANN training process where the targets in the training samples need to be processed synchronously, the number of training samples contained in each group of training samples can be determined according to the actual situation of the distributed computing system, thereby improving the training efficiency of the ANN.

[0074] As an optional example, when executing S520, the system 100 can divide the M training samples according to the computing capabilities of each computing unit, so that the number of targets processed by each computing unit matches the processing capability of the computing unit, thereby reducing the difference in the time required for each computing unit to process the target.

[0075] For example, if the processing capability of computing unit 0 is relatively weak and the processing capability of computing unit 1 is relatively strong, the processing capabilities of computing unit 0 and computing unit 2 are the same, and the processing capabilities of computing unit 2 and computing unit 3 are the same, then system 100 can divide the 20 training samples into 4 groups according to the above-mentioned method 1 or method 3, wherein computing unit 0 processes the training samples of group 1, computing unit 1 processes the training samples of group 2, computing unit 2 processes the training samples of group 3, and computing unit 3 processes the training samples of group 4.

[0076] For another example, the order of processing capabilities of the four computing units of system 100 is: computing unit 2>computing unit 0>computing unit 1>computing unit 3, then system 100 can divide the 20 training samples into 4 groups according to the above method 2, where computing unit 0 processes the training samples of group 1, computing unit 1 processes the training samples of group 2, computing unit 2 processes the training samples of group 3, and computing unit 3 processes the training samples of group 4.

[0077] For another example, if the processing capabilities of the four computing units of the system 100 are the same, the system 100 may divide the 20 training samples into four training sample groups, each containing 10 targets.

[0078] When the processing capabilities of the various computing units of the system 100 are the same, the system 100 may group the M training samples according to the method described below when executing S520 .

[0079] Arrange M training samples into a sample queue. For any two adjacent training samples in the sample queue, the number of targets contained in the previous training sample is less than or equal to the number of targets contained in the next training sample (i.e., the samples are arranged in order of the number of targets from small to large), or the number of targets contained in the previous training sample is greater than or equal to the number of targets contained in the next training sample (i.e., they are arranged in order from large to small).

[0080] Sequentially divide the sample queue into M / (n·K) groups of training samples, where n is the number of training samples that any one of the K computational units can process at a time, i.e., the number of samples contained in a minibatch.

[0081] According to the extraction rule, M / n training samples are extracted from the M / (n·K) groups of training samples to obtain K groups of training samples, where any group of training samples in the K groups of training samples includes n training samples in any group of training samples in the M / (n·K) groups of training samples, wherein the extraction rule is: n training samples are extracted from each group of training samples in the M / (n·K) groups of training samples each time.

[0082] In the above method, M can be divided evenly by (n·K), and the processor can obtain M training samples according to the values ​​of n and K when executing S510. For example, if n is equal to 1, K is equal to 4, and there are 99 samples in the current data set, 96 samples can be selected from the 99 samples as M training samples.

[0083] Still taking M equal to 20 and K equal to 4 as an example, n can be set to 1, where n is a user-set value. System 100 can sort the 20 training samples according to the number of targets contained in each training sample, either in ascending order or in descending order.

[0084] System 100 sequentially divides the sample queue into five groups of training samples. For example, starting from the beginning of the sample queue, each group of four training samples can be divided into five groups. Each group of training samples can be considered a data chunk, where 5 is the value of M / (n·K). Subsequently, system 100 extracts 20 times (20 is the value of M / n), extracting one training sample each time (1 is the value of n) and inputting it into the queue of the computing unit. Each computing unit's queue must contain at least one training sample from any of the five data chunks. That is, one training sample is extracted from each of the five data chunks and input into the queue of each computing unit.

[0085] The 20 extractions described above can be performed in parallel or serially. For example, if system 100 can simultaneously extract 2n training samples from a data block, where these 2n training samples include n training samples from computational unit 0 and n training samples from computational unit 1, then this behavior is considered two extractions. For another example, system 100 can first extract n training samples from a data block for computational unit 0, and then extract n training samples from the same data block for computational unit 1. This behavior is also considered two extractions.

[0086] Since, in the above scheme, each training sample in the sample queue is arranged according to the number of targets, and each data block is obtained by dividing in sequence, the number of targets contained in different training samples in each data block is equal or approximately equal, and the training samples in each data block are respectively assigned to different computing units. Therefore, the number of targets contained in the training samples processed by each computing unit is equal or approximately equal, which reduces the time difference of the training samples processed by each computing unit. For example, Figure 6 As shown, in the ANN training system to which the method provided by the present application is applied, the time difference in generating gradient values ​​between the computing unit with the fastest computing rate and the computing unit with the slowest computing rate becomes smaller, so that synchronous gradient aggregation can be performed in advance, thereby improving the training efficiency of the ANN.

[0087] The above-mentioned step of extracting training samples can be completed by the input loader thread of the system 100. Each computing unit corresponds to an input loader thread. The input loader threads corresponding to the computing units can extract training samples in parallel or in series.

[0088] Furthermore, this application does not impose any restrictions on the order in which the input loading thread extracts data blocks. The input loading thread may randomly select data blocks to extract training samples, or it may sequentially extract training samples from each data block. Optionally, system 100 may also shuffle the multiple data blocks before the extraction step, i.e., disrupt the order of the data blocks, thereby enhancing the randomness of the training samples processed by each computing unit and preventing overfitting or misfitting in system 100.

[0089] As an optional example, when extracting training samples from multiple data blocks, the system 100 may randomly extract training samples.

[0090] For example, data block 1 contains four training samples, a, b, c, and d. A and B each contain one object, and C and D each contain two objects. These four training samples are arranged as a, b, c, and d according to the number of objects they contain. Data block 2 contains four training samples, e, f, g, and h. E contains two objects, F and G each contain three objects, and D contains four objects. These four training samples are arranged as e, f, g, and h according to the number of objects they contain.

[0091] If extracted in sequence, a and e will be entered into the queue of one computing unit, which processes 3 targets; d and h will be entered into the queue of another computing unit, which processes 6 targets, resulting in a large difference in the number of targets processed by different computing units.

[0092] If randomly selected, a and h may be input into the queue of a computing unit, which processes 5 targets; d and f may be input into the queue of another computing unit, which processes 5 targets. The number of targets processed by each computing unit may be equal or approximately equal, so that the time required for each processing unit to process the target is equal or approximately equal, thereby improving the training efficiency of the ANN.

[0093] As another optional example, the system 100 may read n training samples from the queue of each computing unit through a feeding thread, shuffle the n training samples, and then send them to the computing unit.

[0094] For example, the five training samples in queue 0 contain targets 1, 2, 3, 4, and 5, and each training sample is sent to computing unit 0 in this order; the five training samples in queue 1 contain targets 5, 4, 3, 2, and 1, and each training sample is sent to computing unit 1 in this order. If queues 0 and 1 are not shuffled, the time required for computing unit 1 to process the first training sample will far exceed the time required for computing unit 0 to process the first training sample. If the training samples in queues 0 and 1 are shuffled according to the above method, the number of targets processed by processing unit 0 and processing unit 1 each time may be equal or approximately equal, so that the time required for processing unit 0 and processing unit 1 to process the targets is equal or approximately equal, thereby improving the training efficiency of the ANN.

[0095] It should be noted that the above scheme is only an example, and any method that can divide M training samples into K groups of training samples with equal or approximately equal total target numbers falls within the scope of protection of this application.

[0096] The above describes in detail an example of the ANN training method provided by the present application. It is understandable that, in order to achieve the above functions, the ANN training device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0097] The present application can divide the ANN training device into functional units according to the above method example. For example, each function can be divided into various functional units, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in this application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0098] In the case of an integrated unit, Figure 7 A possible structural diagram of the ANN training device provided by the present application is shown. The device 700 includes: a processing unit 701 and an input unit 702. The processing unit 701 is used to control the device 700 to execute Figure 5 The processing unit 701 may also be used to execute other processes of the technology described herein. The apparatus 700 may also include a storage unit 703 for storing program codes and data of the apparatus 700.

[0099] For example, the processing unit 701 is used to control the input unit 702 to execute:

[0100] M training samples are obtained, where each training sample includes at least one target, and M is an integer greater than or equal to 2.

[0101] The processing unit 701 is further configured to execute:

[0102] According to the targets contained in the M training samples, the M training samples are divided into K groups of training samples. The K groups of training samples correspond one-to-one to K computing units. The K computing units are used to process the targets in the K groups of training samples. Moreover, the K computing units are used to train the ANN based on the processing results of the targets in the K groups of training samples. K is an integer greater than or equal to 2.

[0103] The processing unit 701 can be a processor or controller, for example, a CPU, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The input unit 702 is, for example, a transceiver, and the storage unit 703 can be a memory.

[0104] When the processing unit 701 is a processor, the input unit 702 is a transceiver, and the storage unit 703 is a memory, the ANN training device involved in this application can be Figure 8 The device shown.

[0105] See Figure 8 As shown, the apparatus 800 includes: a processor 801, a communication interface 802 and a memory 803 (optional). The processor 801, the communication interface 802 and the memory 803 can communicate with each other through an internal connection path to transmit control and / or data signals.

[0106] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0107] Therefore, the ANN training device provided in the present application groups training samples based on the targets contained in the training samples. In the ANN training process that requires synchronous processing of the targets in the training samples, the number of training samples contained in each group of training samples can be determined according to the actual situation of the distributed computing system. For example, the number of training samples contained in each group of training samples is equal or approximately equal. In this way, the difference in the time required for different computing units to process targets can be reduced, and the next step of processing that needs to be based on the processing results of all targets can be carried out in advance, thereby improving the training efficiency of the ANN.

[0108] See also Figure 9 , the present application also provides a system architecture 200 for training ANN.

[0109] The server 210 is configured with an input / output (I / O) interface 212 for data interaction with an external device (e.g., client device 230). A "user" can input data into the I / O interface 212 through the client device 230. The data may include, for example, a specified data set (i.e., a training sample set), hyperparameters, and shuffling types.

[0110] The server 210 can call data, codes, etc. in the data storage system 240 , and can also store data, instructions, etc. in the data storage system 250 .

[0111] The processor 211 may use the method 500 to process data (eg, training samples). For specific processing, please refer to the relevant description of the method 500 .

[0112] The training device 220 is used to train the ANN according to the command of the processor 211. The training device 220 is, for example, Figure 1 The individual computing units shown.

[0113] Finally, the I / O interface 212 returns the processing result (eg, the trained ANN) to the client device 240 for the user.

[0114] exist Figure 9 In the case shown in FIG, the user can manually specify the data input into the server 210, for example, in the interface provided by the I / O interface 212 (eg, Figure 4 In another embodiment, the client device 230 can automatically input data to the I / O interface 212 and obtain results. If the automatic data input of the client device 230 requires user authorization, the user can set the corresponding permissions in the client device 230. The user can view the results output by the processor 210 on the client device 230, and the specific presentation form can be, for example, displaying the output results on the screen. The client device 230 can also serve as a data collection terminal to store the collected data (e.g., training samples) in the data storage system 240.

[0115] It is worth noting that Figure 9 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation to the technical solution of the present application. For example, Figure 9 In the embodiment, the data storage system 240 is an external memory relative to the server 210. Alternatively, the data storage system 240 may be placed in the server 210. Similarly, the training device 200 may also be placed in the server 210.

[0116] In each embodiment of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the present application.

[0117] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0118] The steps of the method or algorithm described in conjunction with the disclosure of the present application can be implemented in hardware or by executing software instructions by a processor. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, read-only compact discs (CD-ROMs), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0119] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0120] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application should be included in the scope of protection of this application.

Claims

1. A method for training an artificial neural network, characterized in that include: The processor obtains M training samples, each of the M training samples includes at least one target, and M is an integer greater than or equal to 2; The processor divides the M training samples into K groups of training samples according to the number of targets included in the M training samples and the computing power of the K computing units, the K groups of training samples correspond one-to-one to the K computing units, the K computing units are used to process the targets in the K groups of training samples, and the K computing units are used to train the artificial neural network based on the processing results of the targets in the K groups of training samples, where K is an integer greater than or equal to 2. The dividing the M training samples into K groups of training samples is a first grouping mode, the first grouping mode is one of at least two preset grouping modes, and the second grouping mode of the at least two grouping modes is: dividing the M training samples into K training sample groups with the same number of training samples, The processor divides the M training samples into K groups of training samples according to the number of targets included in the M training samples and the computing capabilities of the K computing units, including: The processor obtains instruction information, where the instruction information is used to instruct selection of the first grouping mode from the at least two grouping modes; The processor divides the M training samples into K groups of training samples based on the indication information, the number of targets included in the M training samples, and the computing capabilities of the K computing units, where the number of targets included in the first group of training samples matches the computing capabilities of the first computing unit, the first group of training samples is any one of the K groups of training samples, and the first computing unit is a computing unit among the K computing units that corresponds to the first group of training samples.

2. The method according to claim 1, characterized in that The processor divides the M training samples into K groups of training samples according to the number of targets included in the M training samples and the computing capabilities of the K computing units, including: The processor arranges the M training samples into a sample queue, wherein, in any two adjacent training samples in the sample queue, the number of targets included in the previous training sample is less than or equal to the number of targets included in the next training sample, or the number of targets included in the previous training sample is greater than or equal to the number of targets included in the next training sample; The processor sequentially divides the sample queue into M / (n·K) groups of training samples, where n is the number of training samples that any one of the K computing units can process at one time, and each of the K computing units can process the same number of training samples at one time; The processor extracts M / n training samples from the M / (n·K) groups of training samples to obtain the K groups of training samples, and any group of training samples in the K groups of training samples includes n training samples in any group of training samples in the M / (n·K) groups of training samples.

3. The method according to claim 2, characterized in that Each of the M / (n·K) groups of training samples is sampled with n training samples each time, including: In the M / (n·K) groups of training samples, n training samples are randomly selected from each group of training samples each time.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The processor shuffles each of the K groups of training samples; The processor sends the K groups of training samples that have been shuffled to the K processing units respectively.

5. An artificial neural network training system, characterized in that: comprising a processor, K computing units and a memory, wherein the processor is configured to execute the method according to any one of claims 1 to 4 based on instructions stored in the memory, to divide the M training samples into the K groups of training samples; The K computing units are used to: process the K groups of training samples to obtain targets of the K groups of training samples; process the targets of the K groups of training samples; and train an artificial neural network based on the processing results of the targets of the K groups of training samples.

6. A device for training an artificial neural network, characterized in that Including input unit and processing unit, The input unit is used to: obtain M training samples, each of the M training samples contains at least one target, and M is an integer greater than or equal to 2; The processing unit is configured to divide the M training samples into K groups of training samples according to the number of targets contained in the M training samples and the computing power of the K computing units, wherein the K groups of training samples correspond one-to-one to the K computing units, and the K computing units are configured to process the targets in the K groups of training samples, and the K computing units are configured to train an artificial neural network based on the processing results of the targets in the K groups of training samples, where K is an integer greater than or equal to 2. The dividing the M training samples into K groups of training samples is a first grouping mode, the first grouping mode is one of at least two preset grouping modes, and the second grouping mode of the at least two grouping modes is: dividing the M training samples into K training sample groups with the same number of training samples, The input unit is further configured to: obtain instruction information, wherein the instruction information is configured to instruct selection of the first grouping mode from the at least two grouping modes; The processing unit is further used to: divide the M training samples into K groups of training samples according to the indication information, the number of targets included in the M training samples, and the computing power of the K computing units, the number of targets included in the first group of training samples matches the computing power of the first computing unit, the first group of training samples is any one of the K groups of training samples, and the first computing unit is a computing unit among the K computing units corresponding to the first group of training samples.

7. The device according to claim 6, characterized in that The processing unit is specifically configured to: Arrange the M training samples into a sample queue, wherein, for any two adjacent training samples in the sample queue, the number of targets included in the previous training sample is less than or equal to the number of targets included in the next training sample, or the number of targets included in the previous training sample is greater than or equal to the number of targets included in the next training sample; Sequentially divide the sample queue into M / (n·K) groups of training samples, where n is the number of training samples that any one of the K computing units can process at one time, and each of the K computing units can process the same number of training samples at one time; Extract M / n training samples from the M / (n·K) groups of training samples to obtain the K groups of training samples, wherein any group of training samples in the K groups of training samples includes n training samples in any group of training samples in the M / (n·K) groups of training samples.

8. The device according to claim 7, characterized in that The processing unit is specifically configured to: In the M / (n·K) groups of training samples, n training samples are randomly selected from each group of training samples each time.

9. The device according to any one of claims 6 to 8, characterized in that The processing unit is further specifically configured to: Shuffling each of the K groups of training samples; The K groups of training samples that have been shuffled are sent to the K processing units respectively.

Images