Neural network training method and device, electronic equipment and storage medium

Abstract

The invention provides a neural network training method and device, electronic equipment and a storage medium, the method is applied to a first processing component, at least one network layer of a neural network is deployed on each processing component, and the method comprises the following steps: in a forward propagation process, processing first training data input into the first processing component in a current sub-batch, the first result data is output, a first activation value of the first processing part in the current sub-batch is stored in a first storage space of the first processing part or a second storage space of the second processing part, the first processing part is in communication connection with the second processing part, and the second processing part is not the first processing part; and in the back propagation process, training the neural network according to the first gradient data input into the first processing part in the current sub-batch and the first activation value read from the first storage space or the second storage space. According to the embodiment of the invention, the storage balance among the processing components in the many-core system can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a neural network training method and device, electronic equipment, and storage medium. Background Art

[0002] As the scale of models continues to expand, the traditional single processor architecture can no longer meet the training needs. Many-core systems can process large-scale data and complex tasks in parallel, providing efficient computing power for application fields such as scientific computing, artificial intelligence, and big data analysis.

[0003] Pipeline parallelism is a common large-scale distributed training strategy for many-core systems, but direct pipeline parallelism can cause the processing cores in the many-core system to be idle for a long time. In related technologies, the idle time of the processing cores is reduced by interleaving the forward propagation process and the reverse propagation process in the pipeline parallel process. However, this method may lead to uneven storage of different processing cores. Summary of the invention

[0004] The present disclosure provides a neural network training method and device based on a many-core system, an electronic device, and a storage medium.

[0005] In a first aspect, the present disclosure provides a neural network training method, which is applied to a first processing component of a many-core system, wherein multiple network layers of the neural network are respectively deployed to multiple processing components of the many-core system, and at least one network layer of the neural network is deployed on each processing component, and the first processing component is one of the multiple processing components, wherein a training batch of the neural network training includes multiple sub-batches, and the first processing component interleaves forward propagation processes and backward propagation processes of different sub-batches in the training batch; for any sub-batch of the training batch, the neural network training method includes: in a forward propagation process, processing first training data input to the first processing component of a current sub-batch, outputting first result data, and storing a first activation value of the first processing component in the current sub-batch in a first storage space of the first processing component or a second storage space of a second processing component, the first processing component is communicatively connected to the second processing component, and the second processing component is one of the multiple processing components other than the first processing component; in a backward propagation process, training the neural network according to first gradient data input to the first processing component of the current sub-batch, and the first activation value read from the first storage space or the second storage space.

[0006] In a second aspect, the present disclosure provides a neural network training device, which is applied to a first processing component of a many-core system, wherein multiple network layers of the neural network are respectively deployed to multiple processing components of the many-core system, and at least one network layer of the neural network is deployed on each processing component, and the first processing component is one of the multiple processing components.

[0007] A training batch of the neural network training includes a plurality of sub-batches, and the first processing component interleavedly performs forward propagation processes and backward propagation processes of different sub-batches in the training batch; the neural network training X device includes: a forward propagation module, for processing, during the forward propagation process, first training data input to the first processing component of the current sub-batch for any sub-batch of the training batch, outputting first result data, and storing the first activation value of the first processing component in the current sub-batch into the first storage space of the first processing component or the second storage space of the second processing component, the first processing component is communicatively connected with the second processing component, and the second processing component is one of the plurality of processing components other than the first processing component; and a backward propagation module, for training the neural network according to the first gradient data input to the first processing component of the current sub-batch and the first activation value read from the first storage space or the second storage space for any sub-batch of the training batch during the backward propagation process.

[0008] In a third aspect, the present disclosure provides an electronic device, comprising: multiple processing cores; and an on-chip network configured to exchange data between the multiple processing cores and external data; wherein one or more instructions are stored in one or more of the processing cores, and one or more of the instructions are executed by one or more of the processing cores, so that one or more of the processing cores can execute the above-mentioned neural network training method.

[0009] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned neural network training method when executed by a processor / processing core.

[0010] The present disclosure provides an embodiment, for any sub-batch of a training batch, in the forward propagation process, the first training data input to the first processing component of the current sub-batch can be processed to output the first result data, and the first activation value of the first processing component in the current sub-batch can be stored in the first storage space of the first processing component or the second storage space of the second processing component; in the reverse propagation process, the neural network can be trained according to the first gradient data input to the first processing component of the current sub-batch and the first activation value read from the first storage space or the second storage space. In the neural network training method provided by the embodiment of the present disclosure, the first activation value generated by the first processing component in the current sub-batch can be stored in the first storage space of the first processing component, or in the second storage space of the second processing component, that is, the second processing component can share the load of the activation value generated by the first processing component, thereby achieving storage balance between the first processing component and the second processing component, which is conducive to improving training efficiency.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 Schematic diagram of the training process of a neural network;

[0014] Figure 2 A schematic diagram of a pipeline parallel process is shown;

[0015] Figure 3 A schematic diagram of a pipeline parallel processing process is shown;

[0016] Figure 4 A schematic diagram of 1F1B pipeline parallelism is shown;

[0017] Figure 5 This is a schematic diagram of data storage for 1F1B parallel pipeline;

[0018] Figure 6 A flowchart of a neural network training method provided by an embodiment of the present disclosure;

[0019] Figure 7 A schematic diagram of a chip array provided in an embodiment of the present disclosure;

[0020] Figure 8 Another schematic diagram of a chip array provided for an embodiment of the present disclosure;

[0021] Fig. 9 A schematic diagram showing storage of a first activation value in an embodiment of the present disclosure is shown;

[0022] Fig.10 A schematic diagram of mapping group division provided in an embodiment of the present disclosure;

[0023] Fig.11 A block diagram of a neural network training device provided in an embodiment of the present disclosure;

[0024] Fig.12 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solutions of the present disclosure, the exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0026] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0027] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "including" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. "Connected" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0030] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are briefly described first.

[0031] Many-core system: Many-core system is a high-performance computing architecture that integrates multiple processors on a single chip and uses parallel computing to improve performance and efficiency. A processing core is an independent processing unit inside a processor. Each processing core can execute its own instruction stream and has its own register set and other hardware resources. With the development of multi-core technology, a processor can include multiple processing cores, so a processor can execute multiple tasks or threads at the same time.

[0032] Neural network training process: The neural network training process mainly includes the forward propagation process and the back propagation process. During the forward propagation, the input data passes through each layer of the neural network in turn, and finally reaches the last layer to generate the prediction result. According to the results of the forward propagation and the expected results, the error under the given loss function can be calculated, and it is used as the gradient to propagate backward from the last layer through each layer of the neural network. The back propagation process is divided into two calculation processes: the activation value gradient data calculation process and the weight gradient data calculation process. Among them, when calculating the activation value gradient data, the chain rule is used to pass the activation value gradient data forward layer by layer. When calculating, the activation value gradient data from the next layer and the weight of the current layer are required to obtain the activation value gradient data from the current layer. When calculating the weight gradient data, the weight gradient data from the next layer and the activation value obtained in the forward calculation of the current layer are required to obtain the weight gradient data of the current layer. The weight gradient data will then be used to update the original weight of the current layer.

[0033] Figure 1 The figure is a schematic diagram of the neural network training process. The neural network training process includes the forward propagation process and the back propagation process. The back propagation process includes the activation value gradient data calculation process and the weight gradient data calculation process. Figure 1 As shown in the figure, the forward propagation process can be called the A process, which is used to calculate the activation value; the activation value gradient data calculation process in the back propagation process can be called the B process, and the weight gradient data calculation process can be called the W process. Among them, A1, A2 and A3 are all forward calculation processes in the forward propagation process, B1, B2 and B3 are activation value gradient calculation processes, and W1, W2, W3 are all weight gradient data calculation processes.

[0034] Pipeline parallelism: Data parallelism, tensor parallelism and pipeline parallelism are commonly used distributed training strategies for neural networks. Pipeline parallelism refers to dividing the neural network into multiple stages (each stage includes at least one network layer of the neural network), and then distributing different stages to different processing components (such as chips, processors, etc.), and each processing component completes the training of the neural network in a relay manner. Specifically, during the forward propagation process, the previous stage will pass the activation value to the next stage; during the backpropagation process, the next stage will pass the activation value gradient data and weight gradient data to the previous stage.

[0035] Figure 2 Figure 2 shows a schematic diagram of the pipeline parallel process. Figure 2 As shown in the figure, the neural network is divided into four stages: Layer0, Layer1, Layer2 and Layer3, which are respectively distributed to four processing components: GPU0, GPU1, GPU2 and GPU3. During the forward propagation process, the activation value is passed from Layer0 to Layer3 in turn; during the backward propagation process, the activation value gradient data and weight gradient data are passed from Layer3 to Layer0 in turn.

[0036] 1F1B pipeline parallelism: 1F1B pipeline parallelism refers to the pipeline parallelism in which the forward calculation process and the backward calculation process are executed alternately. In 1F1B pipeline parallelism, a training batch of a neural network is first split into multiple micro-batches, and then the forward calculation process and the backward calculation process of different micro-batches are executed alternately.

[0037] Pipeline parallelism can improve efficiency, but it will produce many bubbles, that is, the processing components in the many-core system are idle for a long time. Figure 3 The figure shows a schematic diagram of the pipeline parallel processing process. Figure 3 As shown in the figure, the neural network is divided into four stages, which are distributed to four processing components: chip 1, chip 2, chip 3 and chip 4. Figure 3 In , Ai represents the forward calculation process performed by chip i for the current batch, Bi+Wi represents the reverse calculation process performed by chip i for the current batch, where Bi represents the activation value gradient data calculation process of chip i for the current batch, and Wi represents the weight gradient data calculation process of chip i for the current batch. For example, Figure 3As shown, A1 represents the forward calculation process performed by chip 1 for the current batch; A2 represents the forward calculation process performed by chip 2 for the current batch; B4+W4 represents the reverse calculation process performed by chip 4 for the current batch, wherein B4 represents the calculation process of activation value gradient data performed by chip 4 for the current batch, and W4 represents the calculation process of weight gradient data performed by chip 4 for the current batch.

[0038] like Figure 3 As shown, chip 1 generates the activation value of the current batch at A1 and uses the activation value at B1+W1. It can be seen that chip 1 is in an idle state between A1 and B1+W1, and this phenomenon is called bubble generation. Similarly, chip 2 is in an idle state between A2 and B2+W, and chips 3 and 4 also generate bubbles, which will not be repeated.

[0039] In the related art, in order to solve the problem of high bubble rate in pipeline parallelism, a 1F1B pipeline parallelism strategy is proposed. Figure 4 A schematic diagram of 1F1B pipeline parallelism is shown. Figure 4 The 1F1B pipeline parallelism is Figure 3 On the basis of the pipeline parallelism shown, the current batch is divided into four sub-batches.

[0040] exist Figure 4 In , Aij represents the forward calculation process performed by chip i for the jth sub-batch of the current batch, and Bij+Wij represents the reverse calculation process performed by chip i for the jth sub-batch of the current batch, where Bij represents the calculation process of the activation value gradient data of chip i for the jth sub-batch, and Wij represents the calculation process of the weight gradient data of chip i for the jth sub-batch. For example, Figure 4 As shown, A11 represents the forward calculation process performed by chip 1 for the first sub-batch of the current batch, B11+W11 represents the reverse calculation process performed by chip 1 for the first sub-batch of the current batch, A23 represents the forward calculation process performed by chip 2 for the third sub-batch of the current batch, and B44+W44 represents the reverse calculation process performed by chip 4 for the fourth sub-batch of the current batch.

[0041] like Figure 4As shown, chip 1 executes A11, A12, A13 and A14 in sequence. Among them, when chip 1 is executing A12, the activation value generated by chip 1 in A11 has been passed to chip 2, and now chip 2 can execute A21. Similarly, when chip 1 is executing A13, chip 2 can execute A22, and chip 3 can execute A31; when chip 1 is executing 14, chip 2 can execute A23, chip 3 can execute A32, and chip 4 can execute A41. After chip 4 executes A41, the reverse propagation process begins. For chip 1, the activation value generated in A14 can be used by B11+W11 in a shorter time, that is, chip 1 is in an idle state for a shorter time. In particular, the theoretical bubble rate of 1F1B pipeline parallelism can be reduced to 0.

[0042] However, there is storage imbalance between different processing components in 1F1B pipeline parallelism. Figure 5 Schematic diagram of 1F1B parallel pipeline data storage.

[0043] Figure 5 The arrow in represents the generation and use relationship of an activation value. Figure 4 Based on the 1F1B parallel pipeline shown in FIG. 1 , Aij generates activation values, and Bij+Wij uses activation values. Figure 5 As shown, after chip 4 generates an activation value of a sub-batch, it will be consumed by the corresponding reverse calculation immediately. For example, the activation value generated by A42 is immediately consumed by B42+W42, and the activation value generated by A45 is consumed by B45+W45. Therefore, for chip 4, only one activation value needs to be stored. An activation value generated by chip 1 needs to go through an additional three forward and reverse calculation processes before it can be used. For example, the activation value generated by A15 needs to go through three forward and reverse calculation processes (including three reverse calculation processes and three forward calculation processes of interleaved execution) of B12+W12, A16, B13+W13, A17, B14+W14, and A18 before it can be used. Since the activation value needs to occupy storage space when generating and using the activation value, chip 1 needs to store 4 activation values ​​(such as the activation values ​​generated by A15, A16, A17, and A18) at each moment during the interleaved execution process. It can be seen that the storage demand of chip 1 is much greater than that of chip 4, and the storage resources of chip 1 and chip 4 in the many-core system are the same, so the storage resources of chip 1 are relatively tight, while the storage utilization rate of chip 4 is low. In other words, the storage of chip 1 and chip 4 is unbalanced.

[0044] According to the neural network training method of the embodiment of the present disclosure, for any sub-batch of a training batch, in the forward propagation process, the first training data input to the first processing component of the current sub-batch can be processed to output the first result data, and the first activation value of the first processing component in the current sub-batch can be stored in the first storage space of the first processing component or the second storage space of the second processing component; in the reverse propagation process, the neural network can be trained according to the first gradient data input to the first processing component of the current sub-batch and the first activation value read from the first storage space or the second storage space. In the neural network training method provided by the embodiment of the present disclosure, the first activation value generated by the first processing component in the current sub-batch can be stored in the first storage space of the first processing component, or in the second storage space of the second processing component, that is, the second processing component can share the load of the activation value generated by the first processing component, thereby achieving storage balance between the first processing component and the second processing component, which is conducive to improving training efficiency.

[0045] Figure 6 A flowchart of a neural network training method provided by an embodiment of the present disclosure. Figure 6 , for any sub-batch of a training batch of neural network training, the method may include:

[0046] In step S11, during the forward propagation process, the first training data of the first processing component input into the current sub-batch is processed, the first result data is output, and the first activation value of the first processing component in the current sub-batch is stored in the first storage space of the first processing component or the second storage space of the second processing component.

[0047] In step S12, during the back propagation process, the neural network is trained according to the first gradient data of the first processing component input into the current sub-batch and the first activation value read from the first storage space or the second storage space.

[0048] The neural network training method provided by the embodiment of the present disclosure can be applied to the first processing unit of the many-core system. The many-core system may include multiple processing units, and the first processing unit may be one of the multiple processing units of the many-core system. For example, the processing unit may be a chip or board in the many-core system, or any electronic device including a many-core chip or board, the processing unit includes multiple processing cores, and data transmission between processing cores and data transmission between processing cores and external devices are realized through the on-chip network (or routing system) of the many-core system; or, the processing unit may be a processing core in a many-core chip in the many-core system, and the processing core itself includes in-core storage, and different cores are interconnected through the on-chip network.

[0049] The neural network training method provided in the embodiments of the present disclosure can be used to train a neural network. The neural network may include multiple network layers. The multiple network layers of the neural network can be deployed to multiple processing components of the many-core system respectively, and at least one network layer of the neural network is deployed on each processing component. For example, the neural network can be used to perform any one of image processing tasks, speech processing tasks, text processing tasks, and processing tasks.

[0050] According to the neural network training method provided by the embodiment of the present disclosure, a training batch of the neural network training includes multiple sub-batches, and the first processing component interleaves the forward propagation process and the back propagation process of different sub-batches in one training batch. For any sub-batch of a training batch, the first processing component can train the neural network by executing steps S11 and S12.

[0051] In step S11, the first processing component performs a forward propagation process. In the forward propagation process, the first processing component processes the first training data and outputs first result data.

[0052] Among them, the first training data can represent the training data of the current sub-batch input to the first processing component. At least one network layer of the neural network is deployed on the first processing component. After the first training data is input to the first processing component, the network layer deployed on the first processing component processes the first training data and outputs the first result data. The first result data may include at least first feature data (such as a feature map) and a first activation value. Among them, the first feature data represents the first training data and the first activation value represents the feature data and activation value output after passing through the above-mentioned at least one network layer. The first feature data can reflect the key information of the first training data, and the first activation value can reflect the response strength of the network layer to the first training data. Both the first feature data and the first activation value will serve as the input of the next network layer, participate in the calculation of the neural network, and complete the task together.

[0053] After obtaining the first result data, the first processing component may store the first activation value of the current batch into the first storage space of the first processing component or the second storage space of the second processing component.

[0054] The second processing component is one of the multiple processing components of the many-core system that is not the first processing component, and the first processing component is communicatively connected to the second processing component. The first processing component is different from the second processing component, and the first processing component and the second processing component are communicatively connected via an on-chip network of the many-core system. It can be understood that the closer the communication distance between the first processing component and the second processing component is, the more conducive it is to improving the training efficiency of the neural network. For example, when the first processing component is adjacent to the second processing component, the training efficiency of the neural network is higher.

[0055] The first storage space represents the storage space in the first processing unit, and the second storage space represents the storage space in the second processing unit. For example, the first storage space and the second storage space may be static random access memory (SRAM). For example, an SRAM with a read / write width of 16B and a capacity of 12KB may be included. The disclosed embodiment does not limit the type and capacity of the first storage space and the second storage space.

[0056] Considering that in 1F1B pipeline parallelism, for the same training batch, different processing components need to store different numbers of activation values, which causes tight storage resources for some processing components, but low storage resource utilization for other processing components. In step S11, after the first processing component generates the first activation value in the current batch, it can store the first activation value locally or in the second processing component, which can save the storage resources of the first processor and improve the utilization of the storage resources of the second processor, thereby achieving storage balance between the first processor and the second processor.

[0057] In step S12, the first processing component performs a back propagation process. In the back propagation process, the first processing component trains the neural network according to the first gradient data and the first activation value input by the current sub-batch.

[0058] Among them, the first gradient data can represent the gradient data of the current sub-batch input to the first processing component. The first gradient data is transmitted to the first processing component by the next layer of the network layer (referred to as: current layer) deployed by the first processing component. In one example, the first gradient data includes activation value gradient data and weight gradient data. In the current sub-batch, the first processing component can obtain the activation value gradient data transmitted from the current layer to the previous layer based on the activation value gradient data transmitted from the next layer and the weight of the current layer, and obtain the weight gradient data of the current layer based on the weight gradient data transmitted from the next layer and the activation value generated by the current layer (i.e., the first activation value). Afterwards, the first processing component can train the neural network with the activation value gradient data and weight gradient data of the current layer.

[0059] Since the first processing component stores the first activation value in the first storage space or the second storage space in step S11, the first processing component can read the first activation value from the first storage space or the second storage space in step S12. In one example, for any sub-batch, the first processing component stores the first activation value of the current sub-batch in the first storage space during the forward propagation process, and then reads the first activation value from the first storage space during the backward propagation process. In another example, the first processing component stores the first activation value of the current sub-batch in the second storage space during the forward propagation process, and then reads the first activation value from the second storage space during the backward propagation process.

[0060] It should be noted that the activation values ​​generated by the first processing component in different sub-batches of a training batch can be stored in the first storage space or in the second storage space. The detailed storage method will be described later and will not be repeated here.

[0061] According to the neural network training method of the embodiment of the present disclosure, for any sub-batch of a training batch, in the forward propagation process, the first training data input to the first processing component of the current sub-batch can be processed to output the first result data, and the first activation value of the first processing component in the current sub-batch can be stored in the first storage space of the first processing component or the second storage space of the second processing component; in the reverse propagation process, the neural network can be trained according to the first gradient data input to the first processing component of the current sub-batch and the first activation value read from the first storage space or the second storage space. In the neural network training method provided by the embodiment of the present disclosure, the first activation value generated by the first processing component in the current sub-batch can be stored in the first storage space of the first processing component, or in the second storage space of the second processing component, that is, the second processing component can share the load of the activation value generated by the first processing component, thereby achieving storage balance between the first processing component and the second processing component, which is conducive to improving training efficiency.

[0062] The following is a detailed description of the neural network training method according to an embodiment of the present disclosure.

[0063] In some possible implementations, the first processing component and the second processing component may be arranged in a head-to-tail folding manner, that is, the first processing component is adjacent to the last processing component, the second processing component is adjacent to the second-to-last processing component, and so on.

[0064] In one example, Figure 7 Schematic diagram of a chip array provided in an embodiment of the present disclosure. Assume that multiple network layers of a neural network are mapped to 8 chips (chip 1, chip 2, ..., chip 8). Figure 7In the example, C[i,j] is used to represent the chip in the i-th row and j-th column. Chip 1 corresponds to C[1,1] in the chip array, chip 2 corresponds to C[1,2] in the chip array, ..., chip 5 corresponds to C[2,4] in the chip array, ..., and the 8th stage is mapped to C[2,1]. The activation value pipeline is transmitted clockwise from C[1,1] to C[2,1], and the gradient data pipeline is transmitted counterclockwise from C[2,1] to C[1,1]. The forward calculation and reverse calculation in each chip can be referred to Figure 4 , I will not go into details here.

[0065] See also Figure 7 , C[2,1] is directly connected to C[1,1], C[1,1] is the first processing unit, and C[2,1] is the second processing unit. C[1,1] stores the activation values ​​generated by the current sub-batch; the second stage is mapped to C[1,2], and the seventh stage is mapped to C[2,2]. C[2,2] and C[1,2] are directly interconnected, C[1,2] is the first processing unit, and helps store the activation values ​​generated by the current sub-batch; and so on.

[0066] In yet another example, Figure 8 Another chip array schematic diagram provided for an embodiment of the present disclosure. Assume that multiple network layers of a neural network are mapped to 16 chips. Figure 8 In , C[i,j] is used to represent the chip in the i-th row and j-th column. Figure 8 As shown, the activation value pipeline transmission starts from C[1,1] and ends at C[2,1], and the gradient data pipeline transmission starts from C[2,1] and ends at [1,1]. And the first processing unit and the second processing unit are set in a head-to-tail folding manner.

[0067] In the embodiment of the present disclosure, the first processing component and the second processing component are arranged in a head-to-tail folding manner, which can achieve storage balance between the first processing component and the second processing component and save communication resources of the on-chip network of the many-core system.

[0068] In some possible implementations, the number of the multiple processing components included in the many-core system is N, and the multiple network layers of the neural network are deployed in sequence. The first processing component in step S11 may be the nth processing component among the N processing components, and the second processing component may be the N-n+1th processing component among the N processing components, where n and N are positive integers, and 1≤n≤N.

[0069] In 1F1B pipeline parallelism, multiple network layers of the neural network can be divided into N stages and deployed on N processing components in sequence. For the same training batch, the earlier the network layer is deployed on a processing component, the more storage capacity of activation values ​​is required on the processing component; the later the network layer is deployed on a processing component, the less storage capacity of activation values ​​is required on the processing component. For example, refer to Figure 5 , multiple network layers of the neural network are deployed on chip 1, chip 2, chip 3 and chip 4 in order from front to back. For the same training batch, the storage capacity of activation values ​​required on chip 1, chip 2, chip 3 and chip 4 is 4, 3, 2 and 1 respectively.

[0070] Assume that the activation value storage required by the nth processing unit is S n , S N +S1=S N-1 +S2=S N-2 +S3=……. Therefore, the nth processing component and the N-n+1th processing component can be used to jointly store the activation values ​​of the same training batch. In the embodiment of the present disclosure, the nth processing component among the N processing components is used as the first processing component, and the N-n+1th processing component is used as the second processing component. At this time, the sum of the storage amounts of the activation values ​​required by the first processing component and the second processing component is more balanced. The activation values ​​generated by the first processing component during the forward propagation process of the current sub-batch can be stored in the first processing component or in the second processing component, which can save storage resources in the first processing component and improve the utilization rate of storage resources in the second processing component.

[0071] In some possible implementations, the first processing component and the second processing component may be arranged in a non-head-to-tail folding manner, wherein the second processing component, in addition to being able to communicate with the first processing component and not being the first processing component, only needs to have surplus storage resources.

[0072] The number of multiple processing components included in the many-core system is N, and multiple network layers of the neural network are deployed in sequence. The first processing component in step S11 can be the nth processing component among the N processing components, and the second processing component can be one of the N / 2+1th to Nth processing components among the N processing components; when n>N / 2, the second processing component can be one of the 1st to N / 2th processing components among the N processing components; n and N are positive integers, 1≤n≤N.

[0073] As mentioned above, for the same training batch, the closer the network layer is deployed on a processing component, the more storage capacity of activation values ​​required on the processing component; the closer the network layer is deployed on a processing component, the less storage capacity of activation values ​​required on the processing component. In the disclosed embodiment, the many-core system includes N processing components, wherein the network layers deployed on the 1st to N / 2th processing components are relatively close to the front, and the network layers deployed on the N / 2+1th to Nth processing components are relatively close to the back. For the N / 2+1th to Nth processing components, the amount of storage capacity of activation values ​​they need to store is relatively small, so these processing components can serve as the second processing components.

[0074] In one example, the first processing component is the nth processing component, and when n≤N / 2, the second processing component may be N / 2+n processing components. Assuming N is 10, the first processing component is 1 processing component, and the second processing component may be the 6th processing component; the first processing component is the 2nd processing component, and the second processing component may be the 7th processing component; ...; the first processing component is the 5th processing component, and the second processing component is the 10th processing component.

[0075] In the embodiment of the present disclosure, the first processing component and the second processing component are arranged in a non-head-to-tail folding manner, which not only achieves storage balance between the first processing component and the second processing component, but also improves the flexibility of processing component deployment in the many-core system.

[0076] In some possible implementations, the plurality of processing components are arranged in an array, adjacent processing components are communicatively connected, and the first processing component and the second processing component are adjacent to each other.

[0077] In order to save the communication resources of the on-chip network of the many-core system and improve the efficiency of data transmission between the first processing component and the second processing component, thereby improving the training efficiency of the neural network, in an embodiment of the present disclosure, the multiple processing components of the many-core system are arranged in the form of an array, and the first processing component and the second processing component are set in adjacent positions.

[0078] In some possible implementations, storing the first activation value of the first processing component in the current sub-batch to the first storage space of the first processing component or the second storage space of the second processing component in step S11 may include: when n≤N / 2, storing the first activation value to the first storage space and / or the second storage space; when n>N / 2, storing the first activation value to the first storage space of the first processing component. It should be noted that " / " in the embodiments of the present disclosure represents a rounding down operation.

[0079] In the case of n≤N / 2, it indicates that the first processing component is deployed at a relatively early network layer in the neural network, the first processing component requires a large amount of activation value storage, and the storage resources of the first processing component are relatively tight. Correspondingly, the second processing component is deployed at a relatively late network layer in the neural network, the second processing component requires a small amount of activation value storage, and the storage resources of the second processing component are relatively sufficient. Therefore, the first processing component can store the first activation value of the current sub-batch in the first processing component or the second processing component. Among them, when the first processing component stores the first activation value of the current sub-batch in the second processing component, it can save the storage resources of the first processing component and improve the utilization rate of the storage resources of the second processing component.

[0080] In the case of n>N / 2, it indicates that the first processing component deploys a later network layer in the neural network, the first processing component requires a smaller amount of activation value storage, and the storage resources of the first processing component are relatively sufficient. Correspondingly, the second processing component deploys a later network layer in the neural network, the second processing component requires a larger amount of activation value storage, and the storage resources of the second processing component are relatively tight. Therefore, the first processing component can store the first activation value of the current sub-batch in the first processing component, and will not store the first activation value of the current sub-batch in the second processing component. At this time, the activation value generated by the second processing component in the current batch can be stored in the first processing component to save the storage resources of the second processing component and improve the utilization rate of the storage resources of the first processing component.

[0081] In some possible implementations, the number of the multiple processing components is N, the first processing component is the nth processing component among the N processing components, the second processing component is the N-n+1th processing component among the N processing components, and the nth processing component is adjacent to the N-n+1th processing component; the multiple network layers of the neural network are divided into N stages, and the N stages are continuously deployed to the N processing components in a sequence in the neural network.

[0082] Since the first processing component is the nth processing component, the second processing component is the N-n+1th processing component, and the nth processing component is adjacent to the N-n+1th processing component in the many-core system, the first processing component and the second processing component are adjacent processing components in the many-core system, and the communication distance between the first processing component and the second processing component does not need to pass through other processing components, and the communication distance is short, which can effectively improve the training efficiency.

[0083] As mentioned above, in the case of n≤N / 2, the first activation value may be stored in the first storage space and / or the second storage space. This step is described in detail below.

[0084] In some possible implementations, when n≤N / 2, storing the first activation value in the first storage space of the first processing component or the second storage space of the second processing component may include: when n≤N / 2 and the remaining storage resources of the first storage space are greater than a preset threshold, storing the first activation value in the first storage space; when n≤N / 2 and the remaining storage resources of the first storage space are less than or equal to the preset threshold, storing the first activation value in the second storage space.

[0085] The preset threshold value can be used to determine whether the remaining storage resources of the first storage space are sufficient. The preset threshold value can be set as needed, for example, the preset threshold value can be 5% of the total storage resources of the first storage space, or the preset threshold value can be 1KB, etc. The present disclosure embodiment does not limit the preset threshold value.

[0086] When n≤N / 2 and the remaining storage resources of the first storage space are greater than a preset threshold, it indicates that the storage capacity of the activation value required by the first processor is large and the remaining storage resources of the current first storage space are sufficient, so the first processor can store the first activation value in the first storage space. In this way, by storing the first activation value locally, the time for storing and reading the first activation value can be saved, and the communication resources of the on-chip network can be saved.

[0087] When n≤N / 2 and the remaining storage resources of the first storage space are less than or equal to the preset threshold, it indicates that the first processor needs a large amount of storage for the activation value but the remaining storage resources of the current first storage space are tight, so the first processor can store the first activation value in the second storage space. In this way, by storing the first activation value externally, the storage resources of the first processor can be saved, and the utilization rate of the storage resources of the external processor can be improved.

[0088] In some possible implementations, the first processing component needs to store N-n+1 first activation values. Figure 5 , the storage amount of the first activation value required by chip 1 is 4, and the storage amount of the first activation value required by chip 2 is 3. If N-n+1≤N / 2, the number of first activation values ​​stored in the first storage space is N-n+1; if N-n+1>N / 2, the number of first activation values ​​stored in the first storage space is (N+1) / 2, and the number of first activation values ​​stored in the second storage space is (N-2n+1) / 2.

[0089] When N-n+1≤N / 2, it indicates that the first processing component needs a smaller amount of storage for the first activation value. In this case, the first processing component can store the first activation value in the first storage space after generating the first activation value in each sub-batch, without storing the first activation value in the second storage space.

[0090] In the case of N-n+1>N / 2, it indicates that the first processing component requires a larger amount of storage for the first activation value. At this time, after the first processing component generates the first activation value in each sub-batch, it can determine whether to store the currently generated first activation value in the first storage space according to the number of first activation values ​​already stored in the first storage space. If the number of first activation values ​​already stored in the first storage space is less than (N+1) / 2, the first processor can store the currently generated first activation value in the first storage space. If the number of first activation values ​​already stored in the first storage space is equal to (N+1) / 2, the first processor can store the currently generated first activation value in the second storage space. Among them, the number of first activation values ​​that can be stored in the second storage space is (N-2n+1) / 2.

[0091] It can be understood that the first processing component needs to store N-n+1 first activation values. These N-n+1 first activation values ​​are all stored in the first storage space when N-n+1≤N / 2, and when N-n+1>N / 2, (N+1) / 2 first activation values ​​are stored in the first storage space, and (N-2n+1) / 2 first activation values ​​are stored in the second storage space.

[0092] In the embodiment of the present disclosure, the number of first activation values ​​stored in the first storage space and the number of first activation values ​​stored in the second storage space can be determined according to the number of first activation values ​​required by the first processing component, and the number of activation values ​​stored in the first storage space and the second storage space can be balanced.

[0093] In some embodiments, the first processing component needs to store N-n+1 first activation values, which is equivalent to setting a storage queue with a length of N-n+1 for the first processing component, and the storage queue is used to store the first activation value. When N-n+1≤N / 2, the storage queue is entirely located in the first storage space. When N-n+1>N / 2, (N-2n+1) / 2 of the storage queue is located in the first storage space, and the rest is located in the second storage space.

[0094] Fig. 9 A schematic diagram showing storage of the first activation value in an embodiment of the present disclosure is shown. Fig. 9 is Figure 7 It is formed on the basis of the many-core system shown.

[0095] like Fig. 9 As shown, assume that C[1,1] is the first processing unit and C[2,1] is the second processing unit. C[1,1] corresponds to chip 1 in the many-core system, that is, the first processing unit. At this time, N=8, n=1, N-n+1=8, and N / 2=4. It can be seen that N-n+1>N / 2, so (N+1) / 2 (i.e., 4) first activation values ​​in the storage queue of C[1,1] are stored in C[1,1], and (N-2n+1) / 2 (i.e., 3) first activation values ​​in the storage queue of C[1,1] are stored in C[2,1].

[0096] like Fig. 9 As shown, assume that C[1,3] is the first processing unit and C[2,3] is the second processing unit. C[1,3] corresponds to chip 3 in the many-core system, that is, the third processing unit. At this time, N=8, n=3, N-n+1=6, and N / 2=4. It can be seen that N-n+1>N / 2, so (N+1) / 2 (i.e. 4) first activation values ​​in the storage queue of C[1,3] are stored in C[3,1], and (N-2n+1) / 2 (i.e. 3) first activation values ​​in the storage queue of C[1,3] are stored in C[2,3].

[0097] like Fig. 9 As shown, assume that C[2,1] is the first processing unit and C[1,1] is the second processing unit. C[2,1] corresponds to chip 8 in the many-core system, that is, the eighth processing unit. At this time, N=8, n=8, N-n+1=1, and N / 2=4. It can be seen that N-n+1<N / 2, so N-n+1 (i.e., 1) first activation values ​​in the storage queue of C[2,1] are stored in C[2,1].

[0098] like Fig. 9 As shown, assume that C[2,3] is the first processing unit and C[1,3] is the second processing unit. C[2,3] corresponds to chip 6 in the many-core system, that is, the sixth processing unit. At this time, N=8, n=6, N-n+1=3, N / 2=4. It can be seen that N-n+1<N / 2, so N-n+1 (i.e. 3) first activation values ​​in the storage queue of C[2,3] are stored in C[2,3].

[0099] In some possible implementations, the first gradient data in step S12 may include activation value gradient data and weight gradient data. The first processing component may include multiple processing units, and the multiple processing units are divided into a first mapping group, a second mapping group, and a third mapping group, wherein the first mapping group may be used to obtain a first activation value, the second mapping group may be used to obtain activation value gradient data, and the third mapping group may be used to obtain weight gradient data. The processing unit here may be a processing core in a multi-core chip in a multi-core system, or a multi-core chip in a multi-core system.

[0100] In the embodiment of the present disclosure, the first processing component can simultaneously execute process A, process B and process W in neural network training. Among them, the processing unit executing process A can be called a first mapping group, the processing unit executing process B can be called a second mapping group, and the processing unit executing process C can be called a third mapping group.

[0101] In some possible implementations, the storage of the first activation value is performed by the first mapping group; and the reading of the first activation value is performed by the third mapping group.

[0102] Fig.10 Schematic diagram of mapping group division provided by the embodiment of the present disclosure. Fig.10 As shown, a core represents a processing unit; the first processing unit includes multiple cores, and the multiple cores are divided into a first mapping group, a second mapping group, and a third mapping group; the second processing unit also includes multiple cores. The first mapping group of the first processing unit can store the first activation value to the second processing unit. The third mapping group of the first processing unit can read the first activation value from the second processing unit. Fig.10 As shown, internal data transmission can also be performed between the cores in the second processing component, which will not be described in detail in the embodiment of the present disclosure.

[0103] In one example, the weights may be stored in the first mapping group and / or the second mapping group B. After the second mapping group calculates the activation value gradient data, it may be immediately copied to the third mapping group for subsequent use, or sent to the third mapping group in real time when the third mapping group calculates.

[0104] When the remaining storage resources of the first storage space are greater than a preset threshold, the first mapping group can store the generated activation value in the first storage space, in which case the third mapping group can read the activation value from the first storage space. When the remaining storage resources of the first storage space are less than or equal to the preset threshold, the first mapping group can store the generated activation value in the second storage space, in which case the third mapping group can read the activation value from the second storage space (see Fig.10 ).

[0105] In some possible implementations, the first processing component may further include a storage unit, and the first storage space is arranged in the storage unit.

[0106] In some possible implementations, the first processing component may further include a routing unit, where the routing unit is used to communicate with other processing components.

[0107] For ease of understanding, the interleaving execution process involved in the embodiments of the present disclosure is described below.

[0108] In some possible implementations, the number of the multiple processing components is N, and the multiple network layers of the neural network are deployed in sequence. The first processing component is the nth processing component among the N processing components. The number of the sub-batches is M, M≥N, and the first processing component interleaves forward propagation processes and backward propagation processes of different sub-batches in the training batch, including: continuously executing the forward propagation process of the 1st to Nnth batches; for the mth sub-batch, when (Nn)+1≤m≤M, interleaving the forward propagation process of the mth sub-batch and the backward propagation process of the m-(Nn)th sub-batch, where m is a positive integer; after executing the forward propagation process of the Mth sub-batch and the backward propagation process of the M-(Nn)th sub-batch, continuously executing the backward propagation processes from the M-(Nn)+1th to the Mth sub-batch.

[0109] by Figure 7 For example, N = 4, M = 9. When the first processing unit is chip 1, firstly execute 3 forward propagation processes from A11 to A13, then execute A14, B11, A15, B12 to A19, B16 alternately, and finally execute 3 reverse propagation processes from B17 to B19.

[0110] In the disclosed embodiment, by dividing the training batch into multiple sub-batches and performing the forward propagation process and the back propagation process of different sub-batches alternately, the bubble rate can be effectively reduced and the efficiency of neural network training can be improved.

[0111] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0112] In addition, the present disclosure also provides a neural network training device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any neural network training method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to in the corresponding records of the method part and will not be repeated here.

[0113] Fig.11 A block diagram of a neural network training device provided in an embodiment of the present disclosure.

[0114] Reference Fig.11The embodiment of the present disclosure provides a neural network training device, which is applied to a first processing component of a many-core system, wherein multiple network layers of the neural network are respectively deployed to multiple processing components of the many-core system, and at least one network layer of the neural network is deployed on each processing component, and the first processing component is one of the multiple processing components, wherein a training batch of the neural network training includes multiple sub-batches, and the first processing component interleaves forward propagation processes and back-propagation processes of different sub-batches in the training batch;

[0115] The neural network training device 20 comprises:

[0116] a forward propagation module 21, configured to process, for any sub-batch of the training batch, first training data input to the first processing component of the current sub-batch during the forward propagation process, output first result data, and store the first activation value of the first processing component in the current sub-batch into a first storage space of the first processing component or a second storage space of a second processing component, wherein the first processing component is communicatively connected with the second processing component, and the second processing component is one of the plurality of processing components other than the first processing component;

[0117] The back propagation module 22 is used to train the neural network for any sub-batch of the training batch according to the first gradient data of the first processing component input into the current sub-batch and the first activation value read from the first storage space or the second storage space during the back propagation process.

[0118] Fig.12 A block diagram of an electronic device provided in an embodiment of the present disclosure.

[0119] Reference Fig.12 An embodiment of the present disclosure provides an electronic device, which includes multiple processing cores 1001 and an on-chip network 1002, wherein the multiple processing cores 1001 are connected to the on-chip network 1002, and the on-chip network 1002 is used to exchange data between the multiple processing cores and external data.

[0120] Among them, one or more instructions are stored in one or more processing cores 1001, and the one or more instructions are executed by one or more processing cores 1001, so that one or more processing cores 1001 can execute the above-mentioned neural network training method.

[0121] In some embodiments, the electronic device may be a brain-like chip, which can use vectorized computing and needs to load the weight information and other parameters of the neural network model through an external memory such as a double data rate (DDR) synchronous dynamic random access memory. Therefore, the batch processing used in the embodiments of the present disclosure has a higher computing efficiency.

[0122] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned neural network training method when executed by a processor / processing core. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0123] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned neural network training method.

[0124] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0125] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A neural network training method, characterized in that: A first processing component applied to a many-core system, wherein the plurality of network layers of the neural network are respectively deployed to the plurality of processing components of the many-core system, each processing component is deployed with at least one network layer of the neural network, and the first processing component is one of the plurality of processing components, A training batch of the neural network training includes a plurality of sub-batches, and the first processing component interleaves forward propagation processes and backward propagation processes of different sub-batches in the training batch; For any sub-batch of the training batch, the method comprises: In the forward propagation process, the first training data of the current sub-batch input to the first processing component is processed, the first result data is output, and the first activation value of the first processing component in the current sub-batch is stored in the first storage space of the first processing component or the second storage space of the second processing component, the first processing component is communicatively connected with the second processing component, and the second processing component is one of the multiple processing components other than the first processing component; During the back propagation process, the neural network is trained according to the first gradient data of the first processing component input into the current sub-batch and the first activation value read from the first storage space or the second storage space.

2. The method according to claim 1, characterized in that The number of the plurality of processing components is N, and the plurality of network layers of the neural network are sequentially deployed, the first processing component is the nth processing component among the N processing components, and the second processing component is the N-n+1th processing component among the N processing components, where n and N are positive integers, and 1≤n≤N; The step of storing the first activation value of the first processing component in the current sub-batch into the first storage space of the first processing component or the second storage space of the second processing component comprises: When n≤N / 2, storing the first activation value in the first storage space and / or the second storage space; When n>N / 2, the first activation value is stored in the first storage space of the first processing component.

3. The method according to claim 1, characterized in that The number of the plurality of processing components is N, and the plurality of network layers of the neural network are sequentially deployed, the first processing component is the nth processing component among the N processing components, and the second processing component is one of the N / 2+1th to Nth processing components among the N processing components; n and N are positive integers, 1≤n≤N; The step of storing the first activation value of the first processing component in the current sub-batch into the first storage space of the first processing component or the second storage space of the second processing component comprises: When n≤N / 2, storing the first activation value in the first storage space and / or the second storage space; When n>N / 2, the first activation value is stored in the first storage space.

4. The method according to claim 2 or 3, characterized in that: When n≤N / 2, storing the first activation value in a first storage space of the first processing component and / or a second storage space of the second processing component comprises: When n≤N / 2 and the remaining storage resources of the first storage space are greater than a preset threshold, storing the first activation value in the first storage space; When n≤N / 2 and the remaining storage resources of the first storage space are less than or equal to the preset threshold, the first activation value is stored in the second storage space.

5. The method according to claim 2 or 3, characterized in that: The first processing component needs to store N-n+1 first activation values; If N-n+1≤N / 2, the number of first activation values ​​stored in the first storage space is N-n+1; If N-n+1>N / 2, the number of first activation values ​​stored in the first storage space is (N+1) / 2, and the number of first activation values ​​stored in the second storage space is (N-2n+1) / 2.

6. The method according to claim 1, characterized in that The first gradient data includes activation value gradient data and weight gradient data, the first processing component includes multiple processing units, and the multiple processing units are divided into a first mapping group, a second mapping group and a third mapping group, wherein the first mapping group is used to obtain the first activation value, the second mapping group is used to obtain the activation value gradient data, and the third mapping group is used to obtain the weight gradient data.

7. The method according to claim 6, characterized in that The storing of the first activation value is performed by the first mapping group; The reading of the first activation value is performed by the third mapping group.

8. The method according to claim 1, characterized in that The number of the multiple processing components is N, and the multiple network layers of the neural network are deployed in sequence. The first processing component is the nth processing component among the N processing components. The number of the sub-batches is M, M≥N, and the first processing component interleaves forward propagation processes and backward propagation processes of different sub-batches in the training batch, including: Continuously perform the forward propagation process from the 1st to the Nnth batch; For the mth sub-batch, when (Nn)+1≤m≤M, the forward propagation process of the mth sub-batch and the backward propagation process of the m-(Nn)th sub-batch are performed alternately, where m is a positive integer; After performing the forward propagation process of the Mth sub-batch and the backward propagation process of the M-(Nn)th sub-batch, the backward propagation processes from the M-(Nn)+1th to the Mth sub-batch are performed successively.

9. The method according to claim 1, characterized in that: The plurality of processing components are arranged in an array, adjacent processing components are communicatively connected, and the first processing component and the second processing component are adjacent to each other.

10. The method according to claim 2, characterized in that The number of the multiple processing components is N, the first processing component is the nth processing component among the N processing components, the second processing component is the N-n+1th processing component among the N processing components, and the nth processing component and the N-n+1th processing component are adjacent, n and N are positive integers, 1≤n≤N; the multiple network layers of the neural network are divided into N stages, and the N stages are continuously deployed to the N processing components according to the sequence in the neural network.

11. A neural network training device, characterized in that: A first processing component applied to a many-core system, wherein the plurality of network layers of the neural network are respectively deployed to the plurality of processing components of the many-core system, each processing component is deployed with at least one network layer of the neural network, and the first processing component is one of the plurality of processing components, A training batch of the neural network training includes a plurality of sub-batches, and the first processing component interleaves forward propagation processes and backward propagation processes of different sub-batches in the training batch; The device comprises: a forward propagation module, configured to, for any sub-batch of the training batch, process the first training data input to the first processing component of the current sub-batch during the forward propagation process, output first result data, and store the first activation value of the first processing component in the current sub-batch into the first storage space of the first processing component or the second storage space of the second processing component, wherein the first processing component is communicatively connected with the second processing component, and the second processing component is one of the plurality of processing components other than the first processing component; A back propagation module is used to train the neural network for any sub-batch of the training batch according to the first gradient data of the first processing component input into the current sub-batch and the first activation value read from the first storage space or the second storage space during the back propagation process.

12. An electronic device, characterized in that: include: Multiple processing cores; as well as A network on chip configured to exchange data between the multiple processing cores and external data; wherein one or more instructions are stored in one or more of the processing cores, and one or more of the instructions are executed by one or more of the processing cores, so that one or more of the processing cores can execute the neural network training method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the neural network training method according to any one of claims 1 to 10.

Images