An accelerator supporting multi-layer neural network inter-layer parallel processing

Abstract

The application provides an accelerator supporting inter-layer parallel processing of a multi-layer neural network, comprising: a computing array; a first rearranger for obtaining, as rearranged data, intra-layer and / or inter-layer data in the multi-layer neural network stored in one or more data blocks and capable of being used for subsequent computation, and rearranging the data into a rearranged data matrix of a preset rearrangement granularity, and obtaining network parameters required for processing the rearranged data matrix and rearranging the network parameters into a rearranged weight matrix; an extension module for obtaining the preset rearrangement granularity corresponding to the rearranged data matrix when the computing array processes the rearranged matrix, instructing each processing unit in the computing array to obtain data in the rearranged data matrix and corresponding weights in the rearranged weight matrix according to the preset rearrangement granularity, and processing the data and the weights; and a second rearranger for, after the computing array completes processing of the rearranged data matrix, rearranging results according to results obtained by processing each data in a result original form to restore output of a network layer.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of neural network processors, in particular to the field of hardware acceleration of neural network model computation, and more particularly to an accelerator supporting inter-layer parallel processing of multi-layer neural networks. BACKGROUND

[0002] With the deployment of neural network models in the fields of robot vision, natural language processing, medical imaging, etc., the prediction effect of various tasks has been significantly improved compared to traditional methods. However, the enormous computational load brought by neural network models hinders the deployment of their algorithms on terminal devices with limited computing power and energy consumption resources. In response to this, neural network accelerators have emerged, which are a typical accelerator for accelerating neural network model algorithms. Accelerators have various application forms, such as: embedding the accelerator in the terminal, which is embedded in the terminal device with high energy efficiency to complete parallel operation; or connecting the accelerator as an external device to the terminal to accelerate the offloaded neural network tasks of the terminal; or setting up an accelerator in the cloud, and in existing research, various accelerators are set up in the cloud to accelerate batch inference and operation of neural networks.

[0003] In the parallel computation of neural networks, there is a commonly used inter-layer parallel method, i.e., running multiple network layers in the accelerator at the same time to improve the utilization rate of the operation units of the accelerator. The existing method for implementing inter-layer parallelism is limited by a restriction condition, which is the dependency relationship between layers and the dependence of the current layer on the output of the previous layer. In this case, the layers and the previous layer cannot be parallel, where parallel refers to a coarse-grained parallelism method based on layers as the basic parallel unit.

[0004] Inter-layer parallelism generally occurs in two scenarios: one is the multi-branch case of residual structure, and the multiple parallel branches (network layers) in the middle of a residual structure have no data dependency relationship and can be completely parallel; the other is batch data inference, where each batch has no data dependency relationship, thus completing the parallelism of the network.

[0005] In actual application scenarios, the occurrence of the above two cases is relatively harsh. On the one hand, the occurrence of multi-parallel branch structure is a small probability event, and on the other hand, multi-batch inference brings inconvenience and constraints to users and cache overhead. The current classic inter-layer scheduling needs to occupy a large amount of intermediate cache due to the coarse-grained limitation, and when the intermediate feature vector is too large, the cache cannot accommodate it, so part of the data needs to be temporarily placed in external storage, thus bringing huge memory access overhead and making it difficult to guarantee its advantages. SUMMARY

[0006] Therefore, the purpose of the present application is to overcome the defects of the prior art, and to provide an accelerator supporting inter-layer parallel processing of a multi-layer neural network.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] According to a first aspect of the present application, an accelerator supporting inter-layer parallel processing of a multi-layer neural network is provided, which can be used to process a multi-layer neural network task including a plurality of network layers offloaded from a general-purpose computer, comprising: a computing array for performing single-layer data processing and inter-layer data parallel processing in the multi-layer neural network; a first rearranger for obtaining, as rearranged data, intra-layer and / or inter-layer data in the multi-layer neural network stored on one or more data blocks and capable of being used for subsequent calculation, and rearranging the rearranged data into a rearranged data matrix of a preset rearrangement granularity, and obtaining network parameters required for processing the rearranged data matrix and rearranging them into a rearranged weight matrix; an extension module for obtaining the preset rearrangement granularity corresponding to the rearranged data matrix when the computing array processes the rearranged matrix, instructing each processing unit in the computing array to obtain data in the rearranged data matrix and corresponding weights in the rearranged weight matrix according to the preset rearrangement granularity for processing; and a second rearranger for, after the computing array completes processing of the rearranged data matrix, rearranging the results according to the original form of the results output by the corresponding layer in the multi-layer neural network to restore the output of the network layer.

[0009] In some embodiments of the present application, the accelerator further comprises a data management module for associating and centrally storing the one or more rearranged data matrices and their corresponding rearranged weight matrices, the preset rearrangement granularity, and the rearranged data of the input data whose rearrangement is completed in the corresponding data block.

[0010] In some embodiments of the present application, the accelerator further comprises a scheduling decision maker for dynamically adjusting the preset rearrangement granularity according to hardware architecture information and data processing delay of the accelerator.

[0011] In some embodiments of the present application, the hardware architecture information includes the hierarchy of the accelerator, the space size of each cache, the data block size of the cache, and the memory bandwidth.

[0012] In some embodiments of the present application, the scheduling decision maker comprises: a search space generator configured to generate a search space according to the hardware architecture information and the data in the current multi-layer neural network that has been available for subsequent calculation, wherein the search space comprises a plurality of scheduling modes, each of which comprises a preset reorganization granularity, reorganized data composition, and storage locations of the original data blocks corresponding to each reorganized data; and a mode generator configured to generate a scheduling mode according to the search space, and collect historical data corresponding to the generated scheduling mode to guide generation of a new scheduling mode that minimizes processing latency, wherein the historical data comprises resource utilization of each cache and processing latency of the corresponding reorganized data when the corresponding scheduling mode is applied to the accelerator.

[0013] In some embodiments of the present application, the accelerator comprises a global cache, a weight cache, and a data cache, the cache hierarchy indicates the hierarchy and connection relationship between the global cache, the weight cache, the data cache, and the computing array, and the memory bandwidth comprises the access bandwidth between each component having a connection relationship in the global cache, the weight cache, the data cache, and the computing array.

[0014] In some embodiments of the present application, each row of the reorganized data matrix corresponds to a row in which a convolution kernel of a convolution position is unfolded, and the data in each row of the reorganized data matrix is convoluted using the weights in the corresponding row of the reorganization weight matrix.

[0015] In some embodiments of the present application, the preset reorganization granularity is smaller than the granularity of the original matrix in which the reorganized data is located.

[0016] In some embodiments of the present application, the extension module comprises: a wide register configured to store the width of each row to indicate that the current computing array needs to read the corresponding data and weights according to the width of each row in the preset reorganization granularity for operation; a row register configured to store the total number of rows of the reorganized data matrix; an accumulation register configured to store the number of rows of the current reorganized data matrix that have been processed by the computing array; an adder configured to, after the computing array completes the calculation of each row of data, add one to the value stored in the accumulation register to complete an accumulation operation, and update the value stored in the accumulation register; and a comparator configured to compare the values of the accumulation register and the row register to determine whether the calculation of the reorganized data matrix of the preset reorganization granularity is completed.

[0017] According to a second aspect of the present application, a computing method executed on the accelerator of the first aspect is provided, comprising: S1, obtaining a multi-layer neural network task from a general-purpose computer; S2, scheduling the multi-layer neural network task in an inter-layer parallel manner, wherein a scheduling decision maker determines a scheduling mode that minimizes processing time delay according to hardware architecture information of the current accelerator and historical data, wherein the hardware architecture information comprises a hierarchy of the accelerator, a space size of each cache, a data block size of the cache, a memory access bandwidth, and the historical data comprises resource utilization of each cache and processing delay of corresponding reorganized data when the historical scheduling mode is applied on the accelerator; S2, reorganizing, by a first reorganizer, intra-layer and / or inter-layer data that has been available for subsequent computation as reorganized data according to a preset reorganization granularity indicated in the current scheduling mode into a reorganized data matrix of the preset reorganization granularity, obtaining network parameters required for processing the reorganized data matrix, rearranging the network parameters into a reorganized weight matrix, and generating a control code; S3, controlling, by an expansion module, each processing unit in a computing array to obtain data in the reorganized data matrix and corresponding weights in the reorganized weight matrix according to the preset reorganization granularity for processing, and storing the result in the cache and initiating an interrupt after the processing is completed, and waiting for processing of the next reorganized data matrix; S4, reorganizing, by a second reorganizer, the result according to the result obtained by processing each data into a form of original result output by a corresponding layer in the multi-layer neural network to restore the output of the network layer after the computing array processes the reorganized data matrix; and S5, repeating steps S2-S4, and outputting the processed result to the general-purpose computer after the multi-layer neural network task is processed.

[0018] Compared with the prior art, the present application has the following advantages:

[0019] The present application provides an accelerator supporting inter-layer parallel processing of a multi-layer neural network, which reorganizes intra-layer and / or inter-layer data that has been available for subsequent computation and corresponding weights into a reorganized data matrix and a corresponding reorganized weight matrix of a smaller granularity than the original matrix of the reorganized data by a first reorganizer, and instructs a computing array to work according to a preset reorganization granularity to complete corresponding processing by an expansion module, and reorganizes the result according to the result obtained by processing each data into a form of original result output by a corresponding layer in the multi-layer neural network to restore the output of the network layer by a second reorganizer. Thus, intra-layer and / or inter-layer data that has been available for subsequent computation and corresponding weights in the neural network are reorganized in a finer granularity, reducing the situation that subsequent computation cannot be carried out due to unprepared available data, more efficiently implementing processing of the neural network task, reducing processing delay, and improving the acceleration effect of the accelerator. BRIEF DESCRIPTION OF DRAWINGS

[0020] The embodiments of the present application will be further described below with reference to the accompanying drawings, in which:

[0021] Figure 1A schematic diagram of the principle of inter-layer parallel processing according to an embodiment of the present application;

[0022] Figure 2 A schematic diagram of the module of an accelerator according to an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the module structure of an accelerator according to an embodiment of the present application;

[0024] Figure 4 A schematic diagram of an expansion module according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0026] As mentioned in the background section, the current classic inter-layer scheduling needs to occupy a large amount of intermediate cache due to the coarse-grained limitation, and when the intermediate feature vector is too large, it needs to temporarily place part of the data in the external storage, which will bring huge memory access overhead and make it difficult to guarantee its advantages. In view of this, the present application proposes an accelerator supporting inter-layer parallel processing of multi-layer neural networks, which reorganizes the data and corresponding weights that can be used for subsequent calculation within and / or between layers into a reorganized data matrix and corresponding reorganized weight matrix with smaller granularity than the original matrix of the reorganized data by a first reorganizer, and instructs the calculation array to work according to the preset reorganization granularity by an expansion module to complete the corresponding processing, and the results of the processing are reorganized by a second reorganizer according to the original form of the results output by the corresponding layer in the multi-layer neural network to restore the output of the network layer. Thus, the data within and / or between layers in the neural network that can be used for subsequent calculation and the corresponding weights are reorganized with finer granularity, reducing the situation that subsequent calculation cannot be carried out due to unprepared available data, more efficiently realizing the processing of neural network tasks, reducing processing delay, and improving the acceleration effect of the accelerator.

[0027] For illustration, see Figure 1For example, it is assumed that part of the data of the Lth layer network layer and the L+1th layer network layer of the neural network is ready for subsequent calculation; however, if the data of the Lth layer or the L+1th layer is not fully ready from the perspective of a single layer, it is difficult to carry out subsequent calculation with the prior art. In this regard, to solve the problem, the application provides an accelerator supporting inter-layer data parallel processing of a multi-layer neural network, which has a fine-grained inter-layer parallel neural network accelerator architecture. By analyzing the dependency relationship between the front and rear layers, the data in the layer and / or between the layers that can be used for subsequent calculation is unfolded into row data, thereby splitting the original inter-layer parallelism into inter-row parallelism. Each time, the rows that need to be parallel processed are selected for processing, so as to realize more fine-grained parallelization scheduling and obtain the improvement of the calculation efficiency.

[0028] According to an embodiment of the application, an accelerator supporting inter-layer data parallel processing of a multi-layer neural network is provided, which is shown in Figure 2 which comprises a computing array 101, a first rearranger 102, an expansion module 103, a second rearranger 104, a scheduling decision maker 105, and a data management module 106, or a combination thereof, which are introduced respectively as follows.

[0029] According to an embodiment of the application, the computing array 101 is used to perform single-layer data processing and inter-layer data parallel processing in a multi-layer neural network. The computing array 101 contains a plurality of processing units for implementing multiplication and accumulation operation on the input single-layer data or inter-layer data and the corresponding weights. The computing array 101 can adopt the existing structure. Alternatively, the computing array 101 comprises a multiplication unit and an adder. The scale of the multiplication unit can be selected as a typical configuration such as 8 by 8, 64 by 64, 128 by 128, etc. The adder is divided into a wide-bit adder and a low-bit adder, which can be used to implement addition of data with different bit numbers.

[0030] According to an embodiment of the present application, the first rearranger 102 is configured to obtain the intra-layer and / or inter-layer data stored in the plurality of layers of the neural network in the one or more data blocks as rearranged data and rearrange the rearranged data into a rearranged data matrix with a preset rearranged granularity, and obtain the network parameters required for processing the rearranged data matrix and rearrange the network parameters into a rearranged weight matrix. Each row of the rearranged data matrix corresponds to a row of the convolution kernel unfolded at a convolution position, and the data in each row of the rearranged data matrix is convoluted using the weights in the corresponding row of the rearranged weight matrix. The granularity of the convolution position can be defined according to the needs of the implementer, for example: one convolution position can be defined as the area covered by the convolution kernel on a single channel; or alternatively, one convolution position can be defined as the area covered by the convolution kernel on a predetermined number of channels (such as 2 channels, 3 channels, etc.). The first rearranger 102 can rearrange the inter-layer data of different layers into a rearranged data matrix, so as to reduce the data preparation time and avoid the situation that the data of the next layer cannot be processed due to the incomplete preparation of the data. It should be understood that in some cases, the data to be processed of a single layer can also fill a rearranged data matrix, and therefore, there can be a case that only the intra-layer data is rearranged to form a rearranged data matrix.

[0031] According to an embodiment of the present application, the extension module 103 is configured to obtain the preset rearranged granularity corresponding to the rearranged data matrix when the computing array processes the rearranged matrix, and instruct each processing unit in the computing array to obtain the data in the rearranged data matrix and the corresponding weights in the rearranged weight matrix according to the preset rearranged granularity for processing. After rearrangement, the embodiment instructs the size of the preset rearranged granularity of the rearranged matrix after rearrangement by the extension module 103 to indicate how to obtain the data in the rearranged data matrix and the corresponding weights in the rearranged weight matrix to complete the convolution operation.

[0032] According to an embodiment of the present application, the second rearranger 104 is configured to, after the computing array completes the processing of the rearranged data matrix, rearrange the results according to the results obtained by processing each data in the original form of the results output by the corresponding layer in the multi-layer neural network to restore the output of the network layer. The results obtained after the processing of the rearranged data matrix and the rearranged weight matrix are different from the arrangement of the original output of the network layer, and therefore, the second rearranger 104 is used to rearrange the results to restore the output of the network layer.

[0033] According to an embodiment of the present application, the scheduling decision maker 105 is configured to dynamically adjust the preset reorganization granularity according to hardware architecture information and data processing delay of the accelerator. The hardware architecture information includes hierarchy of the accelerator, space size of each cache, data block size of the cache, and memory access bandwidth. Preferably, the scheduling decision maker 105 includes: a search space generator configured to generate a search space according to the hardware architecture information and data in the current multi-layer neural network that has been available for subsequent computation within and / or between layers, wherein the search space includes a plurality of scheduling modes, each of which includes a preset reorganization granularity, reorganized data composition, and storage location of each reorganized data corresponding to an original data block; and a mode generator configured to generate a scheduling mode according to the search space and collect historical data corresponding to the generated scheduling mode to guide generation of a new scheduling mode that minimizes processing delay, wherein the historical data includes resource utilization of each cache and processing delay of corresponding reorganized data when the corresponding scheduling mode is applied to the accelerator.

[0034] According to an embodiment of the present application, the data management module 106 is configured to associate and store one or more reorganized data matrices and corresponding reorganized weight matrices of the rearranged input data and the preset reorganization granularity in corresponding data blocks. The data management module 106 can manage transmission and storage of data.

[0035] According to an embodiment of the present application, the accelerator further includes a global cache, a weight cache, and a data cache, the hierarchy indicates the hierarchy and connection relationship between the global cache, the weight cache, the data cache, and the computing array, and the memory access bandwidth includes access bandwidth between each component having a connection relationship in the global cache, the weight cache, the data cache, and the computing array. The global cache can store all data related to data processing of the multi-layer neural network, including all weights, user input, and intermediate results of processing. The weight cache includes a plurality of high-speed storage units and is an on-chip cache used to cache weight parameters. The data cache includes an input cache and an output cache; the input cache is used to store input data and includes a plurality of high-speed storage units and is an on-chip cache used to cache input data; the output cache is used to store output data and includes a plurality of high-speed storage units and is an on-chip cache used to cache output data, such as intermediate results or final results.

[0036] Referring to Figure 3 , an exemplary accelerator structure is described below.

[0037] According to an embodiment of the present application, an accelerator supporting inter-layer data parallel processing of a multi-layer neural network includes:

[0038] Global cache 301, used to store all data related to data processing of the multi-layer neural network, including all weights, user input, and intermediate results of processing.

[0039] Weight cache 302, including several high-speed storage units, is an on-chip cache used to cache weight data, input from the global cache, and output connected to the calculation array 306 to participate in the corresponding operation of the multi-layer neural network.

[0040] Input cache 303, including several high-speed storage units, is an on-chip cache used to cache input data, input from the global cache, and output connected to the selector 305.

[0041] Output cache 304, including several high-speed storage units, is an on-chip cache used to cache output data, input from the calculation array 306 and the output of the activation function unit 308, and communication connection to the calculation array 305, the accumulation register 307, the activation function unit 308, and the global cache 301.

[0042] Calculation array 305, including several processing units, is used to implement multiplication and accumulation operations on input single-layer data or inter-layer data and their corresponding weights. The calculation array 305 can adopt existing structures. Alternatively, the calculation array 101 includes multiplication units, and the scale of the multiplication units can be selected from typical configurations such as 8 by 8, 64 by 64, 128 by 128, etc. In addition to the basic multiplication units, the calculation array 101 includes at least one logic operation unit, not limited to 'and', 'or', 'not', 'exclusive or', 'comparison','shift' operation units, for performing logic operations on input data; includes at least one adder unit, which can perform at least one type of bit addition operation, for performing addition operations on input data; and the calculation array needs to perform accumulation and calculation during the calculation process, so the output of the calculation array needs to be connected to the accumulation register. The calculation array can be connected to the data connection management unit for controlling the input and output of data from the corresponding cache to the output cache by the data connection management unit; when including multiple logic operation units, it can also include a gating unit for implementing one of the logic operations.

[0043] Accumulation and register 307, accumulation register is used to save the intermediate result of the calculation array 305 operation, written by the calculation array, when the accumulation number reaches the expected, it can be written back to the output cache or handed over to the next level of activation function unit 308 to complete the calculation. Accumulation and register 307 contains at least one bit type of several register units, which also contains a gate module inside, which is used to realize the selection of the register, responsible for storing the intermediate result of matrix operation, when the accumulation and operation is completed, the result can be directly written back to the high-speed output cache 304. In this application, the matrix operation mainly refers to the convolution operator, which can be direct convolution operation or im2col.

[0044] Activation function unit 308, used to complete the calculation of nonlinear activation function, including but not limited to ReLU function, binary activation operation function, Sigmoid function, Tanh function, LeakyReLU function, PReLU function, RRELU function, Hardsigmoid function, Hardswish function, ELU function, Mish function or its combination. When containing multiple activation units, it also contains a gating unit for selecting the operation of the required activation function. The output of the activation function unit 308 is connected to the output cache 304 to store the intermediate result obtained by the activation function unit 308 to the output cache 304.

[0045] Data management module Figure 3 (not shown), used to associate and store one or more reorganized data matrices and their corresponding reorganized weight matrices, whose preset reorganization granularity is completed, in the corresponding data block (such as the data block of global cache, weight cache, input cache).

[0046] Referring to Figure 4 , which shows an illustrative extension module, according to an embodiment of the application, the extension module includes a wide register 401, a row register 402, an accumulation register 403, an adder 404 and a comparator 405, wherein:

[0047] Wide register 401, used to store the width of each row to indicate that the current calculation array needs to read the corresponding data and weight according to the width of each row in the preset reorganization granularity for operation; it contains at least one bit type of several register units, the value of which can be dynamically set by the runtime software according to the preset reorganization granularity corresponding to each reorganized data matrix, indicating the width of a row of data, and the calculation array obtains the width of a row of data through the register, calculates the offset address, obtains the data and completes the calculation; wherein the runtime software is a control module that dynamically calls the hardware resources on the accelerator to complete the operation of one or more reorganized data matrices;

[0048] a row register 402, configured to store the total number of rows of the reorganized data matrix; the value of the row register 402 can be dynamically set by the runtime software according to the preset reorganization granularity corresponding to each reorganized data matrix;

[0049] an accumulation register 403, configured to store the number of rows of the current reorganized data matrix that has been processed by the computing array; the accumulation register 403 includes a plurality of register units of at least one bit type, and the initial value of the accumulation register 403 is 0; the accumulation register 403 is updated by an adder 404; and the output of the accumulation register 403 is connected to a comparator 405;

[0050] the adder 404, configured to, after the computing array completes the calculation of one row of data, add 1 to the value stored in the accumulation register 403 to complete an accumulation operation, and update the value stored in the accumulation register 403. For example, the adder 404 generates a trigger signal after the computing array completes the calculation of one row of data, and the adder 404 adds 1 to the value stored in the accumulation register 403 to complete an accumulation operation, and updates the value stored in the accumulation register 403 in response to the trigger signal.

[0051] the comparator 405, configured to compare the value of the accumulation register 403 and the value of the row register 402 to determine whether the calculation of the reorganized data matrix of the preset reorganization granularity is completed, wherein when the value of the accumulation register 403 is greater than or equal to the value of the row register 402, it is determined that the calculation of the current reorganized data matrix is completed. That is, the comparator is responsible for comparing the size of two input values, and the inputs of the comparator 405 are the accumulation register 403 and the row register 402 respectively; when the value of the accumulation register 403 is greater than or equal to the value of the row register 402, the output of the comparator is 1, which is sent to the runtime software as an interrupt signal, and the runtime software can schedule the calculation task of the next reorganized data matrix.

[0052] According to one embodiment of the present application, a method for performing computation on an accelerator is provided, the method comprising: S1, obtaining a multi-layer neural network task from a general-purpose computer; S2, scheduling the multi-layer neural network task in an inter-layer parallel manner, wherein a scheduling decision maker determines a scheduling mode that minimizes processing latency according to hardware architecture information of the current accelerator and historical data, wherein the hardware architecture information comprises a hierarchy of the accelerator, a space size of each cache, a data block size of the cache, a memory bandwidth, and the historical data comprises resource utilization of each cache when a corresponding historical scheduling mode is applied on the accelerator and processing latency of corresponding reorganized data; S2, reorganizing, by a first reorganizer, data within a layer and / or between layers that have been available for subsequent computation as reorganized data according to a preset reorganization granularity indicated in the current scheduling mode into a reorganized data matrix of the preset reorganization granularity, obtaining network parameters required for processing the reorganized data matrix and rearranging into a reorganized weight matrix, and generating a control code; S3, controlling, by an expansion module according to the control code, each processing unit in a computation array to obtain data in the reorganized data matrix and corresponding weights in the reorganized weight matrix according to the preset reorganization granularity for processing, and storing the result in a cache and initiating an interrupt after the processing is completed, and waiting for processing of the next reorganized data matrix; S4, reorganizing, by a second reorganizer, the result according to a form in which the result is originally output by a corresponding layer in the multi-layer neural network after the computation array completes processing of the reorganized data matrix, to restore the output of the network layer; S5, repeating steps S2-S4, and outputting the processed result to the general-purpose computer when the multi-layer neural network task is completed.

[0053] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be performed in the above specific order, and in fact, some of the steps can be performed concurrently or even in a changed order, as long as the desired function can be achieved.

[0054] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present application.

[0055] A computer readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor, or any other suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0056] Having described above several embodiments, now will be described a number of modifications and alternatives. Such description is included to provide for a more complete understanding of the various embodiments and is not intended in any way to limit the scope of possibilities. Many modifications in addition to those described above can be made by one skilled in the relevant art without departing from the scope and spirit of the described embodiments. The scope of the various embodiments disclosed herein cover all technical solutions falling within the principles and their practical applications or improvements to the art, as well as enabling other ordinary skilled in the art to understand the various embodiments disclosed herein.

Claims

1. An accelerator supporting inter-layer parallel processing of a multi-layer neural network, having a fine-grained inter-layer parallel neural network accelerator architecture, by analyzing the dependency between the previous and next layers, unfolding the intra-layer and inter-layer data that has been available for subsequent computation into row data, thereby splitting the original inter-layer parallelism into inter-row parallelism, selecting the rows that need to be processed in parallel each time, achieving a more fine-grained parallelization scheduling, which can be used to process a multi-layer neural network task including multiple network layers offloaded from a general-purpose computer, comprising: a computing array for performing single-layer data processing and inter-layer data parallel processing in the multi-layer neural network; a first rearranger for obtaining the intra-layer and inter-layer data that has been available for subsequent computation in the multi-layer neural network stored on one or more data blocks as rearranged data and rearranging the rearranged data into a rearranged data matrix of a preset rearrangement granularity smaller than the granularity of the matrix where the rearranged data originally resides, and obtaining the network parameters required for processing the rearranged data matrix and rearranging the network parameters into a rearranged weight matrix; an extension module for obtaining the preset rearrangement granularity corresponding to the rearranged data matrix when the computing array processes the rearranged matrix, instructing each processing unit in the computing array to obtain data in the rearranged data matrix and the corresponding weights in the rearranged weight matrix according to the preset rearrangement granularity for processing; a second rearranger for, after the computing array completes processing of the rearranged data matrix, rearranging the results according to the original form of the results output by the corresponding layer in the multi-layer neural network to restore the output of the network layer. The accelerator further comprises a data management module for associating and storing the one or more rearranged data matrices and their corresponding rearranged weight matrices, the preset rearrangement granularity, and the rearranged data after rearrangement of the input data in the corresponding data blocks. The accelerator further comprises a scheduling decision maker for dynamically adjusting the preset rearrangement granularity according to hardware architecture information of the accelerator and data processing delay. The hardware architecture information includes the hierarchy of the accelerator, the space size of each cache, the data block size of the cache, and the memory bandwidth. The scheduling decision maker comprises:

2. The accelerator of claim 1, wherein a search space generator for generating a search space according to the hardware architecture information and the intra-layer and inter-layer data that has been available for subsequent computation in the current multi-layer neural network, wherein the search space includes a plurality of scheduling modes, each scheduling mode including a preset rearrangement granularity, rearranged data composition, and storage location of the original data block corresponding to each rearranged data; 3. The accelerator of claim 1, wherein a mode generator for generating a scheduling mode according to the search space and collecting historical data corresponding to the generated scheduling mode to guide the generation of a new scheduling mode that minimizes processing delay, wherein the historical data includes resource utilization of each cache and processing delay of the corresponding rearranged data when the corresponding scheduling mode is applied on the accelerator.

4. The accelerator of claim 3, wherein The accelerator includes a global cache, a weight cache, and a data cache, a cache hierarchy indicates the hierarchy and connection relationship between the global cache, the weight cache, the data cache, and the computing array, and a memory bandwidth includes the access bandwidth between each component having a connection relationship in the global cache, the weight cache, the data cache, and the computing array.

5. The accelerator of claim 4, wherein, ​ ​ ​ 6. The accelerator of claim 4, wherein, ​ 7. Accelerator according to one of claims 1-6, characterized in that Each row of the reorganization data matrix corresponds to a row on which a convolution kernel of a convolution position is spread, and data of each row of the reorganization data matrix is subjected to a convolution operation using weights of a corresponding row of the reorganization weight matrix.

8. Accelerator according to one of claims 1-6, characterized in that The extension module comprises: a wide register configured to store a width of each row to indicate that the current computing array needs to read corresponding data and weights according to the width of each row in the preset reorganization granularity for operation; a row register configured to store a total number of rows of the reorganization data matrix; an accumulation register configured to store a number of rows of the current reorganization data matrix that have been processed by the computing array; an adder configured to, after the computing array completes the calculation of one row of data, add one to the value stored in the accumulation register to complete an accumulation operation, and update the value stored in the accumulation register; a comparator configured to compare the value of the accumulation register and the value of the row register to determine whether the calculation of the reorganization data matrix of the preset reorganization granularity is completed.

9. A computing method executed on the accelerator of one of claims 1-8, characterized in that, The method comprises: S1, obtaining a multi-layer neural network task from a general-purpose computer; S2, scheduling the multi-layer neural network task in an inter-layer parallel manner, wherein a scheduling decision maker determines a scheduling mode that minimizes processing delay according to hardware architecture information and historical data of the current accelerator, wherein the hardware architecture information comprises a hierarchy of the accelerator, a space size of each cache, a data block size of the cache, a memory bandwidth, the historical data comprises resource utilization of each cache and processing delay of corresponding reorganization data when a corresponding historical scheduling mode is applied to the accelerator; S2, a first rearranger rearranges data within a layer and between layers that have been available for subsequent calculation as reorganization data according to a preset reorganization granularity indicated in the current scheduling mode, and reorganizes the reorganization data into a reorganization data matrix of the preset reorganization granularity, and obtains network parameters required for processing the reorganization data matrix and rearranges the network parameters into a reorganization weight matrix, and generates a control code; S3, an extension module controls each processing unit in the computing array to obtain data in the reorganization data matrix and corresponding weights in the rearranged weight matrix according to the preset reorganization granularity for processing according to the control code, and stores the result in the cache after the processing is completed and initiates an interrupt, and waits for the next reorganization data matrix to be processed; S4, a second reorganizer reorganizes the result according to the original form of the result output by the corresponding layer in the multi-layer neural network after the computing array completes the processing of the reorganization data matrix to restore the output of the network layer; S5, repeating steps S2-S4, and outputting the processed result to the general-purpose computer when the processing of the multi-layer neural network task is completed.

Images