Edge-end neural network acceleration system and edge-end neural network model training method

Abstract

The application discloses an edge-end neural network acceleration system and a training method of an edge-end neural network model, wherein the edge-end neural network acceleration system comprises an input module, a calculation module, and a data receiving module; the input module is used for sending input data needing calculation to the calculation module and sending an instruction to the calculation module according to state information, and the state information is generated according to network information of the neural network; the calculation module comprises a core calculation unit and one or more auxiliary calculation units, is used for acquiring a connection relationship and network information of the core calculation unit and the auxiliary calculation unit according to the instruction, connecting the core calculation unit and the auxiliary calculation unit with each other according to the instruction, and executing a neural network training calculation task according to the input data and the network information, the neural network training calculation task comprises a main calculation task and an auxiliary calculation task, the core calculation unit is used for executing the main calculation task, and the auxiliary calculation unit is used for executing the auxiliary calculation task; and the data receiving module is connected with the calculation module and is used for receiving a calculation result of the calculation module. The execution unit of a task having the same fine-grained operation representation in different training is set as the core calculation unit, so that the core calculation unit can be used for execution in different training, and hardware utilization and energy consumption performance are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a system for processing neural network training, and in particular to an edge-end neural network acceleration system and a training method for an edge-end neural network model. BACKGROUND

[0002] In recent years, artificial intelligence technology based on deep learning algorithms has developed rapidly and is widely used in image recognition, natural language processing, intelligent diagnosis and other fields. The engineering application process of deep learning algorithms, i.e., the process of deep learning algorithm computation from cloud environment to actual edge environment. Referring to Figure 1 , the traditional process of deep learning edge application is to train the network on high-performance hardware in the cloud to obtain the network weight information, and then send the weight information to the edge hardware. The edge hardware performs inference according to the network weight information obtained by the cloud hardware training to complete the actual task required by the deep learning algorithm. However, in the actual application process, due to the differences in data distribution in time or space between the data used for cloud training and the data used for actual task completion at the edge, the accuracy of the deep learning algorithm in completing the task decreases. Referring to Figure 2 , in order to reduce the influence of the data difference between the cloud and the edge on the network weight in the deep learning algorithm, the edge can perform secondary training on the network weight using its own data. At this time, the training process in the cloud is called pre-training, and the training process in the edge is called re-training. The secondary training process improves the problem of mismatch between the data distribution of the cloud and the edge, and improves the accuracy of the deep learning algorithm in completing the task.

[0003] A complete deep learning algorithm can be divided into three parts: forward inference, back propagation, and weight update. Among them, forward inference is the process of deep learning algorithm executing actual tasks according to network weight information, back propagation and weight update are the processes of deep learning algorithm changing network weight information through training data, so that network weight information better reflects data characteristics and improves the accuracy of deep learning algorithm in completing tasks. In the traditional cloud-edge collaboration process, the edge hardware does not need to modify the network weight information, so the edge hardware only needs to execute the forward inference part of the deep learning algorithm. In the secondary training cloud-edge collaboration, the edge device also needs to execute the training of the deep learning algorithm, so it needs to support the three parts of complete forward inference, back propagation and weight update.

[0004] The cloud-side neural network training acceleration system meets the high-throughput and high-concurrency neural network training requirements of the cloud by stacking a large number of computing devices such as GPUs and TPUs (see Norman P.J, George K, Sheng L, et al. TPUv4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings [C] / / 2023 ACM / IEEE 50th Annual International Symposium on Computer Architecture (ISCA), ACM, 2023: 1-14.), but this brings high power consumption and is not suitable for edge-side retraining application scenarios because the sudden small batch of data input on the edge side cannot effectively input all the computing devices.

[0005] Most of the existing edge-side hardware systems focus on optimizing the forward inference part, such as NVDLA (referring to NVIDIA. NVDLA Primer [EB / OL]. 2018: [2024.3.8]. http: / / nvdla.org / primer.html.), Cambricon (referring to Shaoli L, Zidong D, Jinhua T, et al. Cambricon: An Instruction Set Architecture for Neural Networks [C] / / 2016 ACM / IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), ACM, 2016: 393-405.), which cannot calculate the back propagation and weight update. A small number of hardware systems that support complete edge training use a layer-by-layer pipelining architecture such as CNN-MERP (referring to Xushen H, Dajiang Z, Shihao W, et al. CNN-MERP: An FPGA-based memory-efficient reconfigurable processor for forward and backward propagation of convolutional neural networks [C] / / 2016 IEEE 34th International Conference on Computer Design (ICCD), IEEE, 2016: 320-327.) or a heterogeneous core architecture (Choi S, Sim J, Kang M, et al. An energy-efficient deep convolutional neural network training accelerator for in situ personalization on smart devices [J]. IEEE Journal of Solid-State Circuits, 2020, 55(10): 2691-2702.), which results in low hardware utilization. The layer-by-layer pipelining method customizes a core for the forward inference calculation of each layer of the neural network and another core for the back propagation and weight update of each layer of the neural network. With the increase of the number of layers of the neural network, the hardware utilization of this architecture decreases rapidly, because only one core of one layer is involved in operation at the same time.The heterogeneous core architecture takes into account the hardware reusability of computations at different layers of the neural network. It customizes one computation core only for forward inference and another for backpropagation and weight updates. Theoretically, this approach has a hardware utilization rate of no more than 55% (because the two cores are different in size and the neural network algorithm executes forward inference, backpropagation, and weight updates at different times).

[0006] Existing acceleration hardware for deep learning algorithms either cannot perform backpropagation and weight update calculations, or has low hardware utilization efficiency in calculating backpropagation and weight updates, which leads to reduced energy efficiency and prevents the large-scale use of cloud-edge collaborative mode for secondary training. Summary of the Invention

[0007] The purpose of this invention is to solve the problem of improving the hardware utilization efficiency of edge neural network acceleration systems, and to provide an edge neural network acceleration system and a training method for edge neural network models.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] An edge neural network acceleration system, comprising:

[0010] The input module is used to send input data to be calculated to the calculation module, and to send instructions to the calculation module according to the status information, wherein the status information is generated based on the network information of the neural network.

[0011] The computing module includes a core computing unit and one or more auxiliary computing units, and is used to obtain the connection relationship and network information between the core computing unit and the auxiliary computing unit according to the instructions, connect the core computing unit and the auxiliary computing unit to each other according to the instructions, and execute a neural network training computing task according to the input data and network information. The neural network training computing task includes a main computing task and an auxiliary computing task.

[0012] The core computing unit is used to execute the main computing tasks, which are computing tasks with the same fine-grained operation representation in the forward inference stage, the backpropagation stage, and the weight update stage.

[0013] The auxiliary computing unit is used to perform auxiliary computing tasks, which are computing tasks with different fine-grained operation representations in the forward inference stage, the backpropagation stage, and the weight update stage. The number of the auxiliary computing units corresponds to the types of auxiliary computing tasks required by the neural network.

[0014] The data receiving module is connected to the calculation module and is used to receive the calculation results of the calculation module.

[0015] In some embodiments, the core computing unit comprises a plurality of stacked computing sub-units, each of which is capable of simultaneously computing a part of the main computing task and aggregating the computing results.

[0016] In some embodiments, the state information comprises state information indicating the computing stage and an input data address.

[0017] In some embodiments, the main computing task comprises full connection layer computation, convolution computation, and pooling layer computation.

[0018] In some embodiments, the auxiliary computing task comprises activation function computation, bias computation, and data normalization computation.

[0019] In some embodiments, the input module comprises an instruction input storage unit, a control unit, and a data input storage unit, the instruction input storage unit is used to store state information other than specific input data required for the computing module to complete the computing task and send instructions to the computing module, the data input storage unit is used to store and send input data required for computation to the computing module, and the control unit is connected with the instruction input storage unit and used to translate the instructions into hardware signals recognizable by the computing module and send them to the computing module.

[0020] The application also provides a training method for an edge-end neural network model, characterized in that it comprises the following steps:

[0021] S1: the input module sends input data required for computation to the computing module and sends instructions to the computing module according to state information, which is generated according to network information;

[0022] S2: the computing module acquires the connection relationship between the core computing unit and the auxiliary computing unit and the network information according to the instructions, connects the core computing unit and the auxiliary computing unit with each other, and performs neural network training computation in the order of tasks according to the input data and the network information;

[0023] In some embodiments, the core computing unit performs a main computing task according to the network information and the data input, the main computing task is a computing task with the same fine-grained operation representation in the forward inference stage, the backward propagation stage, and the weight update stage, the core computing unit comprises a plurality of stacked computing sub-units, each of which simultaneously computes a part of the main computing task and aggregates the computing results.

[0024] The auxiliary computing unit performs an auxiliary computing task according to the network information and the data input, the auxiliary computing task is a computing task with different fine-grained operation representations in the forward inference stage, the backward propagation stage, and the weight update stage.

[0025] S3: outputting a calculation result, and a data receiving module receiving the calculation result of the calculation module;

[0026] S4: inputting the calculation result as back propagation stage input data, and repeating steps S1-S3 to obtain a back propagation stage output result;

[0027] S5: updating network weights according to the back propagation stage output result.

[0028] In some embodiments, in step S2, the connection relationship of each auxiliary calculation unit is determined according to the sequence relationship between the auxiliary calculation tasks, and the connection relationship between the auxiliary calculation task units and the core calculation unit is determined according to the sequence relationship between the auxiliary calculation tasks and the main calculation task.

[0029] In some embodiments, if the same auxiliary calculation task exists in the forward inference stage and the back propagation stage, the same auxiliary calculation unit can be used in step S2.

[0030] In some embodiments, if the task to be executed is the weight updating stage, step S5 includes the following steps:

[0031] The network weights are updated by the weight updating auxiliary calculation unit, and then step S1 is entered for one cycle to obtain the result output of the neural network model after the weight updating.

[0032] The present application has the following beneficial effects:

[0033] In a conventional acceleration system for edge-side neural network training, the forward inference and the back propagation are distinguished when the hardware module is designed, so that the two have different hardware modules when implemented, and the basis for sharing hardware modules to improve hardware utilization is lacking. The present application comprehensively considers the forward inference stage and the back propagation stage, sets the calculation tasks with the same fine-grained operation representation in the forward inference stage, the back propagation stage and the weight updating stage as main calculation tasks, uses a core calculation unit to execute the main calculation task and an auxiliary calculation unit to execute the auxiliary calculation task, adjusts the connection relationship of the core calculation unit and the auxiliary calculation unit according to the instruction of the input module, and can reuse the core calculation unit in the forward inference stage and the back propagation stage and the weight updating stage, so as to achieve high hardware utilization in the whole stage training.

[0034] Other beneficial effects in the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a schematic diagram of the cloud-edge collaboration principle in the prior art;

[0036] Figure 2 is the cloud edge collaborative principle of secondary training in the prior art;

[0037] Figure 3 is the edge neural network acceleration system structure schematic diagram in the embodiment of the application;

[0038] Figure 4 The specific structure schematic diagram of the edge neural network acceleration system in the embodiment of the application;

[0039] Figure 5 The training method flow chart of the edge neural network model in the embodiment of the application;

[0040] Figure 6 is the core computing array schematic diagram in the embodiment of the application;

[0041] Figure 7 is the task execution sequence schematic diagram when multiple computing tasks need to be executed in the embodiment of the application;

[0042] Figure 8 is the neural network structure schematic diagram that needs to be trained in the embodiment of the application;

[0043] Figure 9 is the forward inference stage training flow chart in the embodiment of the application;

[0044] Figure 10 is the back propagation stage training flow chart in the embodiment of the application. DETAILED DESCRIPTION

[0045] The embodiments of the application are described in detail below. It should be emphasized that the following description is only exemplary, not intended to limit the scope of the application and its applications.

[0046] Reference Figure 3 The edge neural network acceleration system of the embodiment of the application comprises:

[0047] The input module is used for sending the input data needing to be calculated to the computing module, and sending instructions to the computing module according to state information, wherein the state information is generated according to network information of the neural network;

[0048] The computing module comprises a core computing unit and one or more auxiliary computing units, and is used for acquiring the connection relationship and network information of the core computing unit and the auxiliary computing unit according to the instructions, connecting the core computing unit and the auxiliary computing unit with each other according to the instructions, and executing the neural network training computing task according to the input data and the network information, wherein the neural network training computing task comprises a main computing task and an auxiliary computing task;

[0049] The core computing unit is configured to execute a main computing task, which is a computing task having the same fine-grained operation representation in the forward inference stage, the backward propagation stage, and the weight updating stage.

[0050] The auxiliary computing unit is configured to execute an auxiliary computing task, which is a computing task having different fine-grained operation representations in the forward inference stage, the backward propagation stage, and the weight updating stage, and the number of the auxiliary computing units corresponds to the types of the auxiliary computing tasks required by the neural network.

[0051] The data receiving module is connected with the computing module and is configured to receive a computing result of the computing module.

[0052] Reference Figure 4 In the embodiment, the edge neural network acceleration system specifically comprises:

[0053] An instruction input storage unit, which is an instruction input memory in the embodiment, stores all state information required by the control computing array and the auxiliary computing unit to complete a computing task, except for specific input data, and includes state information indicating a computing stage, an input data address, and computing stage state information indicating which stage of the forward inference and the backward propagation the computing task belongs to, so as to control the connection relationship of the computing array and the auxiliary computing unit, and the input data address indicates the specific position of the required input data of the computing task in the data input storage unit, that is, the data input storage unit has a larger storage space than the input data required by a computing task, and is used for data input storage unit data management and correct reading of the input data of the computing module.

[0054] A data input storage unit, which is a data input memory in the embodiment, stores input data actually required for computing;

[0055] A control unit connected with the instruction input storage unit, which is configured to translate the instruction provided by the instruction input memory into a hardware signal recognizable by the computing module and send the hardware signal to the computing module.

[0056] The computing module includes the core computing unit and the auxiliary computing unit, and the core computing unit is a core computing array in the embodiment; the updated model output and the computing completed by the system, the new model weight information obtained after the original model is modified, and the task result output, which is the task result obtained by executing the task by the original model before training, are output to the data receiving module; the core computing unit includes a plurality of stacked computing sub-units, each of which can simultaneously compute a part of the main computing task and aggregate the computing results.

[0057] If the result of the task performed by the updated model is required, the result output of the task performed by the updated model can be obtained by running the system once again after the original training task is completed.

[0058] The purpose of the embodiment of the present application is to propose a hardware system architecture for edge-side neural network training capable of realizing high hardware utilization and high energy efficiency, which mainly utilizes the reuse of core computing units in the forward inference stage and the backward propagation and weight update stage to achieve high hardware utilization in the whole stage. The main innovation point of the embodiment of the present application can be considered as the innovation of the neural network accelerator architecture. The architecture can realize the training of the neural network alone on the edge side, or can be used as edge-side heavy training hardware for cloud-edge collaboration.

[0059] The function of the embodiment of the present application is to complete the training of the neural network model on the edge side. The training of the neural network model includes three parts of forward inference, backward propagation and weight update. The system first disassembles the computing tasks in the three stages into core operations based on convolution and matrix multiplication and other auxiliary calculations. In different stages, core operations need to be performed, but different auxiliary calculations need to be performed. By configuring instructions, the connection relationship between the core computing array and the auxiliary calculation unit is changed in different stages, so as to realize the improvement of hardware utilization and the reduction of computing energy consumption.

[0060] The existing hardware acceleration system for training on the edge side separately processes the three stages of forward inference, backward propagation and weight update. How to correspond the algorithms and hardware modules becomes a key problem. When a hardware module corresponds to an excessively long algorithm flow, the performance of the hardware module itself is increased, but the universality of the hardware module is decreased, thereby affecting the overall utilization of the hardware system. When the part of the algorithm flow accelerated by a hardware module is too short, this kind of calculation method is closer to the way of calculating neural network algorithms by traditional computing devices, thereby leading to the inability to fully utilize the computing parallelism and the reduction of the computing efficiency.

[0061] The prior art considers algorithm splitting as follows: a neural network algorithm is stacked by layers of networks with similar structures, in the forward inference process, hardware calculates sequentially from front to back until the task result is calculated, in training, the task result is calculated by forward inference, then the task result is compared with the training data set to obtain the calculation error, the calculation error is back propagated from back to front layer by layer until each layer network obtains its own error value, and finally the weights of each layer network are updated according to the error value. The pipeline architecture designs a separate hardware module for the forward inference of each layer, and designs a separate hardware module for the back propagation and weight update of each layer, which causes the hardware to be too fragmented, affecting the overall utilization of the hardware system. The heterogeneous core architecture designs a main hardware module for forward inference and another main hardware module for back propagation and weight update, which causes the hardware utilization bottleneck during actual calculation by calling the two modules back and forth.

[0062] The embodiments of the present application consider the algorithm splitting as follows: when we jump out of the consideration of the traditional three computing stages, and view the computing task from a more bottom layer, it can be found that the computing tasks of the three stages have certain commonality, that is, they can all be regarded as main computing tasks with large computing amount and auxiliary computing tasks with small computing amount. For the division of the computing task, in the neural network forward inference stage, according to the existing research, the full connection layer computation, convolution computation and pooling layer computation of the network are divided into main computing tasks with large computing amount, and the main computing task is a task that can find the same fine-grained operation representation. The full connection layer computation, convolution computation and pooling layer can find the same fine-grained operation representation. The high parallelism feature guarantees the system performance, because these computing tasks have the characteristics of large computing amount and high parallelism, and the computing time accounts for more than 90% of the total computing time. At the same time, there is no data dependency relationship between the data in the computing process, and it has very high parallelism (high parallelism is the characteristic of simultaneously performing multiple computing processes. The full connection layer computation, convolution computation and pooling layer can share a core computing unit. The reason is that these computations can be represented by finer-grained operations of multiplication first and addition later. Since the full connection layer computation, convolution computation and pooling layer can be represented by finer-grained operations of multiplication first and addition later, they can share the same computing array. The operation of high parallelism guarantees that the array has a large enough scale to improve the computing energy efficiency and save the computing time). And the activation function computation, bias computation and data normalization computation are divided into auxiliary computing tasks with small computing amount, because these computing processes account for less computing time in the whole network computation process. On this basis, for the back propagation phase of neural network training, we define the error propagation process of the full connection layer computation, convolution computation and pooling layer computation as main computing tasks with large computing amount, and define the error propagation process of the activation function computation, bias computation and data normalization computation as auxiliary computing tasks with small computing amount. The auxiliary computing task is a task that cannot find the same fine-grained operation representation. This division logic is similar to the division logic in the forward inference.

[0063] A core computing array is designed for the main computing task with large computing amount, that is, a stack of computing units, such as Figure 5The number of auxiliary computing units and the connection mode with the core computing array are determined according to specific tasks for small amount of calculation auxiliary computing tasks. For a neural network task, there are multiple auxiliary computing tasks, and the number of auxiliary computing units corresponds to the types of auxiliary computing tasks required by the network. The auxiliary computing tasks required by the network are divided into auxiliary computing tasks required for forward reasoning and auxiliary computing tasks required for back propagation. The connection relationship of each auxiliary computing unit can be determined according to the sequence of auxiliary computing tasks, and the sequence relationship of auxiliary computing tasks and main computing tasks can determine the connection relationship between auxiliary computing units and core computing array. That is, the sequence on the task corresponds to the sequence of the connection of the hardware module. For the forward reasoning and back propagation process, they are independent of each other, have different auxiliary computing units and different connection sequences, but share a core computing array. If the forward reasoning and back propagation process have the same auxiliary computing task, the same auxiliary computing unit can be used, but in different stages of forward reasoning and back propagation, there will be different connection relationships. The computing unit stacked in the core computing array is a floating-point multiply-accumulate circuit (MAC circuit). The hardware of the auxiliary computing unit is different according to the auxiliary computing task it implements, but it can be summarized as a combination of a series of digital circuits that implement addition, subtraction, multiplication, division and shifting.

[0064] Reference Figure 5 The core computing array is characterized by fully utilizing the parallelism of the main computing task. That is, each computing unit can simultaneously compute a part of the complete main computing task, and finally the computing results are summarized, that is, the main computing can be represented by a more fine-grained operation of multiplying first and adding later. Each computing unit can independently compute a multiply-first-and-add-later operation, and the summary result of all multiply-first-and-add-later computing results is the final result of the main computing task. This way reduces the computing time required by the main computing task by increasing the number of hardware. The connection method of the computing unit array in the core computing array is not unique. Common ones are the convolution array mode of the convolution module in the NVDLA neural network accelerator of NVIDIA Company and the systolic array mode in the TPU accelerator of Google Company.

[0065] Reference Figure 6 The training method of the edge neural network model of the embodiment includes the following steps:

[0066] S1: The input module sends the input data to be calculated to the computing module, and sends instructions to the computing module according to the state information, and the state information is generated according to network information;

[0067] S2: The computing module obtains the connection relationship and network information of the core computing unit and the auxiliary computing unit according to the instruction, connects the core computing unit and the auxiliary computing unit with each other, and performs neural network training calculation according to the input data and the network information in the task order;

[0068] The core computing unit performs a main computing task according to the network information and the data input, the main computing task is a computing task with the same fine-grained operation representation in the forward inference stage, the backward propagation stage and the weight update stage, the core computing unit includes a plurality of stacked computing sub-units, each stacked computing sub-unit simultaneously calculates a part of the main computing task, and the calculation results are summarized;

[0069] The auxiliary computing unit performs an auxiliary computing task according to the network information and the data input, the auxiliary computing task is a computing task with different fine-grained operation representations in the forward inference stage, the backward propagation stage and the weight update stage;

[0070] The connection relationship of each auxiliary computing unit is determined according to the order of the auxiliary computing tasks, and the connection relationship between the auxiliary computing task unit and the core computing unit is determined according to the order relationship between the auxiliary computing task and the main computing task;

[0071] S3: Output the calculation result, the data receiving module receives the calculation result of the computing module, and refers to Figure 7 If the neural network training includes a plurality of computing tasks, the main computing task of the previous computing task can be completed, and the main computing task of the subsequent computing task can be performed to improve the training efficiency;

[0072] S4: The result is used as the input data of the backward propagation stage, and steps S1-S3 are cycled to obtain the output result of the backward propagation stage, if the forward inference stage and the backward propagation stage have the same auxiliary computing task, the same auxiliary computing unit can be used in step S2;

[0073] S5: The network weight is updated according to the output result of the backward propagation stage, specifically, the network weight is updated by the weight update auxiliary computing unit, and then step S1 is entered for one cycle to obtain the result output of the task performed by the neural network model after the weight is updated.

[0074] In the embodiment of the present application, in the forward reasoning stage, the network information can be obtained through the state information, the system obtains the input data and the network information (including network weight, bias, and possible activation function information), the core computing array performs convolution operation on the two data, the calculated data is transmitted to the bias auxiliary computing unit, the bias is calculated by the bias auxiliary computing unit, and then the data is transmitted to the activation function auxiliary computing unit for calculation. The calculation result of the activation function auxiliary computing unit is the calculation result of this forward reasoning stage. The reverse propagation has a similar process, the system obtains the error input data and the network information (including network weight, bias, and possible activation function information), the core computing array performs convolution operation on the two data, the calculated data is transmitted to the bias auxiliary computing unit, the derivative is calculated by the derivative auxiliary computing unit, this calculation is one step of network error transmission, then the data is transmitted to the activation function derivative auxiliary computing unit for calculation. The calculation result of the activation function derivative auxiliary computing unit is the calculation result of this reverse propagation stage. When the error calculation of all layers is completed, the network weight is updated by the weight update auxiliary computing unit. The auxiliary computing unit is a general term for various types of auxiliary computing units, including the above-mentioned bias auxiliary computing unit, derivative auxiliary computing unit, and activation function derivative auxiliary computing unit. The bias auxiliary computing unit, derivative auxiliary computing unit, and activation function derivative auxiliary computing unit are hardware modules for realizing certain auxiliary computing tasks required by the network. The functions correspond to the names, that is, the bias auxiliary computing unit is used to realize the bias calculation required by the network, and the essence is a series of addition calculations. The derivative auxiliary computing unit is used to realize the derivative calculation required by the network reverse propagation, and in the convolution, the performance form is that the elements at the same position of two matrices of the same size are multiplied to obtain a new matrix of the same size, and the essence is a series of multiplication calculations. The activation function derivative auxiliary computing unit is used to realize the activation function derivative calculation required by the network reverse propagation. Different network tasks require different activation functions, so their derivative forms are different. A common example is the Relu activation function, and the calculation essence is to compare with 0, which can be realized by a hardware comparator.

[0075] Taking the CNN network as an example, as Figure 8One of the neural network training calculation process, the main activation function is Relu function. Each block in the figure represents a network layer, the input is 32 pixels by 32 pixels by 3 colors. From top to bottom embodies the process of neural network forward inference, 5x5conv2d represents a 2d convolution kernel size of 5x5, after convolution, the Relu activation function operation is carried out, and then the maxpool maximum value pooling operation is carried out, and the three calculation processes are repeated once, and then the linear full connection layer calculation is carried out, and the Relu activation function operation is carried out after calculation, and the process is repeated three times. The input image is disassembled into 3 32 pixel by 32 pixel subgraphs, which are reduced in size to extract features after convolution operation, such as after the first convolution, it becomes 28 pixels by 28 pixels by 6 sub-feature maps, after the second convolution, it becomes 10 pixels by 10 pixels by 16 sub-feature maps, and after the full connection layer integrates all the sub-feature map information, the image class judgment result is obtained, such as after the first full connection layer, 120 results are obtained, after the second full connection layer, 84 results are obtained, and after the third full connection layer, the final 10 class results are obtained. Based on this, the system's calculation module design is as shown in Figure 9 and Figure 10 .

[0076] The main calculation task of the network is the 5x5 convolution task, and based on this, the main calculation array is designed as a pulsatile calculation array with 25x25 multipliers, as shown in Figure 5 to fully utilize the calculation parallelism in the convolution task. In the forward inference process, the bias needs to be calculated after the convolution operation in the convolution layer, and the Relu activation function layer calculation needs to be calculated after the convolution layer calculation, so the data flow design for the forward inference stage is the core calculation array to the bias unit to the Relu unit to the output buffer. Among them, the bias unit and the Relu unit are auxiliary calculation units at this time, and the connection relationship of the calculation array, the bias unit, the Relu unit is the calculation array to the bias unit to the Relu unit, the Relu unit output buffer, and then the core calculation array input buffer is completed. The maximum pooling layer calculation outputs the calculation result once. After completing the maximum pooling layer calculation, it is considered that the forward inference stage is a calculation, and the output result is stored in the output buffer. At this time, the auxiliary calculation unit belonging to the back propagation process is in an inactive state.

[0077] In the back propagation process, the overall process is similar to the forward inference stage, and the error propagation of the convolution layer needs to calculate a reverse convolution, which can be divided into a convolution calculation and an element-wise point multiplication calculation. After completing the error propagation of the convolution layer, the error propagation of the Relu activation function layer needs to be calculated, which requires a Relu activation function derivative calculation. Therefore, the data flow design for the back propagation stage is from the core calculation array to the element-wise point multiplication unit to the Relu derivative calculation unit. The Relu activation function derivative unit outputs a buffer to the core calculation array input buffer to complete the error propagation calculation of the max pooling layer, and outputs the calculation result to the output buffer. At this time, the element-wise point multiplication unit and the Relu derivative unit are auxiliary calculation units, and the connection relationship of the calculation array, the element-wise point multiplication unit and the Relu derivative unit is that the calculation array is connected with the element-wise point multiplication unit, and the element-wise point multiplication unit is connected with the Relu derivative unit. At this time, the auxiliary calculation unit belonging to the forward inference process is in an inactive state.

[0078] This calculation module is an actual instance for a specific network task. It should be pointed out that this hardware system architecture is not limited to implementing this structure for network tasks, and is also feasible for other neural network training tasks. The same fine-grained operation representation calculation task can be found as the main calculation, such as the calculation task that can be represented by the finer-grained operation of multiplication first and addition later. Auxiliary calculation tasks need to be represented by different fine-grained operations. However, the number of hardware resources of the main calculation array should be greater than that of the remaining auxiliary calculation units.

[0079] In the traditional edge-side neural network training acceleration system, forward inference and back propagation are distinguished in hardware module design, so that they have different hardware modules when implemented, lacking the basis for sharing hardware modules to improve hardware utilization. In this design, forward inference and back propagation are comprehensively considered in hardware module design, and the main calculation part with similar characteristics is extracted, and the core calculation array is designed based on this, thereby improving the hardware utilization and energy consumption performance. We implemented this system on a ZCU102 type FPGA according to the convolutional neural network task, with a theoretical calculation performance of 4.35GFLops (FP32 data format), a power consumption of 1.63W, and a calculation delay of 26.68ms for training a Lenet network through 1000 MNIST dataset pictures. As a comparison, the theoretical calculation performance of the layer-by-layer pipelining system implemented on the same type of FPGA is 4.08GFLops (FP32 data format), the power consumption is 1.85W, and the calculation delay is 78.05ms for training a Lenet network through 1000 MNIST dataset pictures.

[0080] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0081] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0082] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0084] The above further describes the present application in conjunction with specific / preferred embodiments, and cannot be deemed to limit the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or variations to the described embodiments, and these substitutions or variations shall be deemed to fall within the protection scope of the present application. In the description of the present application, the description of the terms "an embodiment", "some embodiments", "a preferred embodiment", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are contained in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the case of no mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples. Although the embodiments of the present application and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. An edge-end neural network acceleration system, comprising: The application relates to a neural network training computing device, which comprises the following parts: an input module, which is used for sending input data needing to be calculated to a computing module and sending instructions to the computing module according to state information, wherein the state information is generated according to network information of the neural network; the computing module, which comprises a core computing unit and one or more auxiliary computing units, is used for acquiring connection relationships and network information of the core computing unit and the auxiliary computing units according to the instructions, connecting the core computing unit and the auxiliary computing units with each other according to the instructions, and executing a neural network training computing task according to the input data and the network information, wherein the neural network training computing task comprises a main computing task and an auxiliary computing task; the core computing unit is used for executing the main computing task, and the main computing task is a computing task having the same fine-grained operation representation in a forward inference stage, a backward propagation stage and a weight updating stage; the auxiliary computing unit is used for executing the auxiliary computing task, and the auxiliary computing task is a computing task having different fine-grained operation representations in the forward inference stage, the backward propagation stage and the weight updating stage, and the number of the auxiliary computing units corresponds to the types of the auxiliary computing tasks required by the neural network; a data receiving module is connected with the computing module and is used for receiving a calculation result of the computing module.

2. The edge-end neural network acceleration system of claim 1, wherein, The core computing unit comprises a plurality of stacked computing subunits, each of which can simultaneously calculate a part of the main computing task and collect calculation results.

3. The edge-end neural network acceleration system of claim 1, wherein, The state information comprises state information indicating a calculation stage and an input data address.

4. The edge-end neural network acceleration system of claim 1, wherein, The main computing task comprises full connection layer calculation, convolution calculation and pooling layer calculation.

5. The edge-end neural network acceleration system of claim 1, wherein, The auxiliary computing task comprises activation function calculation, bias calculation and data normalization calculation.

6. The edge-end neural network acceleration system of claim 1, wherein, The input module comprises an instruction input storage unit, a control unit and a data input storage unit, the instruction input storage unit is used for storing state information required for the computing module to complete a calculation task except specific input data and sending instructions to the computing module, the data input storage unit is used for storing and sending input data needing to be calculated to the computing module, and the control unit is connected with the instruction input storage unit and is used for translating the instructions into hardware signals recognizable by the computing module and sending the hardware signals to the computing module.

7. A method for training an edge-end neural network model, the method comprising: The application further relates to a neural network training computing method, which comprises the following steps: S1: an input module sends input data needing to be calculated to a computing module and sends instructions to the computing module according to state information, wherein the state information is generated according to network information; S2: the computing module acquires connection relationships and network information of a core computing unit and an auxiliary computing unit according to the instructions, connects the core computing unit and the auxiliary computing units with each other, and performs neural network training calculation according to the input data and the network information in a task sequence. The core computing unit performs a main computing task according to the network information and the input data, the main computing task is a computing task with the same fine-grained operation representation in a forward inference stage, a backward propagation stage and a weight updating stage, the core computing unit comprises a plurality of stacked computing sub-units, each stacked computing sub-unit simultaneously calculates a part of the main computing task, and the calculation results are summarized; The auxiliary computing unit performs an auxiliary computing task according to the network information and the input data, the auxiliary computing task is a computing task with different fine-grained operation representations in the forward inference stage, the backward propagation stage and the weight updating stage; S3: outputting a calculation result, a data receiving module receives the calculation result of the calculation module; S4: taking the calculation result as input data of the backward propagation stage, and performing steps S1-S3 to obtain output results of the backward propagation stage; S5: updating network weights according to the output results of the backward propagation stage. 8.The method of Claim 7, wherein, In step S2, a connection relationship of each auxiliary computing unit is determined according to a sequence of the auxiliary computing tasks, and a connection relationship between the auxiliary computing task unit and the core computing unit is determined according to a sequence relationship of the auxiliary computing task and the main computing task. 9.The method of Claim 7, wherein, If the same auxiliary computing task exists in the forward inference stage and the backward propagation stage, the same auxiliary computing unit can be used in step S2.

10. The method of Claim 7, wherein, If the task to be performed is the weight updating stage, step S5 comprises the following steps: The network weights are updated by the weight updating auxiliary computing unit, and then step S1 is entered to perform a cycle to obtain a result output of the neural network model after the weights are updated.

Images