Neural network accelerator, data processing device and neural network acceleration method

Abstract

The present application provides a neural network accelerator, comprising a first processing unit and a second processing unit; a first data transfer module in the first processing unit is connected to a second data transfer module in the second processing unit; the first processing unit sends first target data to the second data transfer module via the first data transfer module, and / or the first processing unit receives second target data sent by the second data transfer module via the first data transfer module. The present application also provides a data processing device and a neural network acceleration method.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a neural network accelerator, a data processing device, and a neural network acceleration method. Background Art

[0002] Deep learning algorithms, due to their superior performance, are widely used in fields such as computer vision, speech recognition, natural language processing, and bioinformatics. Neural network accelerators used for deep learning computing can process datasets such as image data to obtain information of interest, such as facial recognition, intrusion detection, liveness detection, direction detection, object classification, and activity counting.

[0003] Currently, neural network accelerators used for deep learning calculations adopt a distributed storage mode. Although it has the advantage of high processing parallelism, the use of a distributed storage mode will limit the flexibility of implementing deep learning algorithms. Summary of the Invention

[0004] Embodiments of the present application provide a neural network accelerator, a data processing device, and a neural network acceleration method.

[0005] The technical solution of this application is achieved as follows:

[0006] The present application provides a neural network accelerator, comprising: a first processing unit and a second processing unit;

[0007] The first data transfer module in the first processing unit is connected to the second data transfer module in the second processing unit;

[0008] The first processing unit sends the first target data to the second data transfer module through the first data transfer module, and / or the first processing unit receives the second target data sent by the second data transfer module through the first data transfer module.

[0009] Optionally, the first processing unit further includes a first data storage module and a first data processing module;

[0010] The first data transfer module is connected to the first data processing module and the first data storage module respectively;

[0011] The first data storage module is used to store data to be processed;

[0012] The first data processing module is used to process the data to be processed to obtain a processing result.

[0013] Optionally, the first data transfer module includes a forwarding submodule;

[0014] The forwarding submodule is connected to the first data storage module, the first data processing module, and the second data transfer module respectively;

[0015] The forwarding submodule is configured to read the processing result and / or the data to be processed; and send the processing result and / or the data to be processed as the first target data to the second data transfer module.

[0016] Optionally, the first data transfer module includes a calculation submodule and a forwarding submodule;

[0017] The operation submodule is connected to the first data storage module, the first data processing module, and the forwarding submodule respectively;

[0018] The forwarding submodule is also connected to the second data transfer module;

[0019] The operation submodule is used to perform operation processing on the processing result and / or the data to be processed to obtain a first operation result;

[0020] The forwarding submodule is configured to send the first operation result as the first target data to the second data transfer module.

[0021] Optionally, the forwarding submodule is further configured to receive the second target data sent by the second data transfer module;

[0022] The operation submodule is further configured to perform operation processing on at least one of the second target data, the data to be processed, and the processing result to obtain a second operation result.

[0023] Optionally, the neural network accelerator further includes a third processing unit;

[0024] The third data transfer module of the third processing unit is connected to the forwarding submodule;

[0025] The forwarding submodule is further configured to send the second operation result to the third data transfer module.

[0026] Optionally, the operation submodule includes at least one unary operation circuit and / or at least one binary operation circuit;

[0027] The operation type of the at least one unary operation circuit and / or the at least one binary operation circuit is determined based on the configuration information.

[0028] Optionally, the second target data includes at least one of the following:

[0029] The data to be processed is stored in the second data storage module of the second processing unit;

[0030] a processing result obtained by processing by the second data processing module in the second processing unit;

[0031] The fourth target data is sent by the fourth processing unit; the fourth data transfer module of the fourth processing unit is connected to the second data transfer module.

[0032] Optionally, the first processing unit and the second processing unit are any two adjacent processing units among the multiple processing units included in the neural network accelerator.

[0033] An embodiment of the present application also provides a data processing device, including the neural network accelerator provided in the above embodiment.

[0034] The embodiment of the present application further provides a neural network acceleration method, which is applied to a neural network accelerator, wherein the neural network accelerator includes a first processing unit and a second processing unit; a first data transfer module in the first processing unit is connected to a second data transfer module in the second processing unit;

[0035] The method comprises:

[0036] controlling the first processing unit to send first target data to the second data transfer module through the first data transfer module, so that the second processing unit processes the first target data;

[0037] And / or, control the first processing unit to receive the second target data sent by the second data transfer module through the first data transfer module, and process the second target data.

[0038] The neural network accelerator provided in the embodiment of the present application may include a first processing unit and a second processing unit; wherein the first data transfer module in the first processing unit is connected to the second data transfer module in the second processing unit; the first processing unit sends first target data to the second data transfer module via the first data transfer module, and / or the first processing unit receives second target data sent by the second data transfer module via the first data transfer module. In other words, the processing unit in the embodiment of the present application may include a data transfer module, which forwards the data of the current processing unit to other processing units via the data transfer module, thereby realizing information sharing between different processing units and improving the flexibility of the deep learning algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the architecture of a neural network accelerator in a related technology provided in an embodiment of the present application;

[0040] Figure 2A schematic diagram of the structure of a neural network accelerator provided in an embodiment of the present application Figure 1 ;

[0041] Figure 3 A schematic diagram of the structure of a neural network accelerator provided in an embodiment of the present application Figure 2 ;

[0042] Figure 4 A schematic diagram of the structure of a neural network accelerator provided in an embodiment of the present application Figure 3 ;

[0043] Figure 5 A schematic diagram of the structure of a neural network accelerator provided in an embodiment of the present application Figure 4 ;

[0044] Figure 6 A schematic diagram of the structure of a neural network accelerator provided in an embodiment of the present application Figure 5 ;

[0045] Figure 7 A schematic diagram of data transmission provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of the structure of a neural network accelerator provided in an embodiment of the present application Figure 6 ;

[0047] Figure 9 A schematic diagram of the structure of a neural network accelerator provided in an embodiment of the present application Figure 7 ;

[0048] Figure 10 A schematic diagram of an exemplary shuffling operation provided in an embodiment of the present application;

[0049] Figure 11 A flowchart of a neural network acceleration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present invention.

[0051] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0052] To facilitate understanding of the technical solutions of the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below. The following relevant technologies can be arbitrarily combined with the technical solutions of the embodiments of the present application as optional solutions, and they all fall within the protection scope of the embodiments of the present application.

[0053] A neural network accelerator for deep learning calculations may include a convolution processing module, a vector processing module, and a storage module. Among them, the convolution processing module may include a multiplication-accumulation array, and with the help of the multiplication-accumulation array, the convolution operation is completed, that is, the convolution processing module can complete data processing with a high computational density. In addition, the vector processing module mainly completes data processing with a low computational density such as pooling, size scaling, feature image splicing, and feature image data rearrangement. The storage module can adopt a hierarchical design. For example, the storage unit can be divided into a working register, a static random access memory (SRAM), a dynamic random access memory (DRAM), etc.

[0054] In practical applications, neural network accelerators can be optimized in terms of power consumption, performance, and area. When dividing the modules in a neural network accelerator, it is important to keep the correspondence between the processing modules and the storage modules as simple and clear as possible. This will facilitate the layout and routing between modules later and will also improve the timing of data processing.

[0055] refer to Figure 1 The schematic diagram of the architecture of a neural network accelerator in the related art is shown. The neural network accelerator for deep learning calculations may include N independent processing units: processing unit 1 to processing unit N. Each processing unit may include a convolution processing module, a vector processing module, and a storage module. In other words, the neural network accelerator in the related art adopts a distributed storage mode, and each operation module has an exclusive storage module. However, in this architecture, there are obstacles to information sharing between the various storage modules, which limits the flexibility of the deep learning algorithm.

[0056] Based on this, an embodiment of the present application provides a neural network accelerator. Specifically, the neural network accelerator may include a first processing unit and a second processing unit; wherein a first data transfer module in the first processing unit is connected to a second data transfer module in the second processing unit; the first processing unit sends first target data to the second data transfer module via the first data transfer module, and / or the first processing unit receives second target data sent by the second data transfer module via the first data transfer module. In other words, the processing unit in the embodiment of the present application may include a data transfer module, which forwards the data of the current processing unit to other processing units via the data transfer module, thereby realizing information sharing between different processing units and improving the flexibility of the deep learning algorithm.

[0057] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0058] An embodiment of the present application provides a neural network accelerator, referring to Figure 2 The structure of the neural network accelerator shown Figure 1 The neural network accelerator provided in the embodiment of the present application may include: a first processing unit 21 and a second processing unit 22.

[0059] In some embodiments, the neural network accelerator may be an accelerator for deep learning calculations, an accelerator for neural network calculations, etc., which is not limited in the embodiments of the present application. The neural network accelerator in the embodiments of the present application may be provided in a data processing device, a data processing chip, or an electronic device, which may be a smartphone, a tablet computer, a personal computer, a server, or an industrial computer, etc., which is not limited in the embodiments of the present application.

[0060] It should be noted that the neural network accelerator may include multiple independent processing units, and multiple processing units can process data in parallel. In the embodiment of the present application, each processing unit may correspond to the data of a channel group, that is, each processing unit can process the data of the corresponding channel group. Here, a channel group may include one channel or multiple channels, and the embodiment of the present application does not limit this. The data of the channel group can be understood as feature information. For example, in image processing applications, the output of the channel can be a feature map of the image to be processed.

[0061] In real-world applications, data transmission and computation are mostly concentrated in a single channel group. Data exchange between channel groups is infrequent, meaning that data exchange between different units is rare. Therefore, having multiple processing units process data from different channel groups can improve data processing parallelism.

[0062] In some embodiments, the first processing unit 21 and the second processing unit 22 may be two different adjacent processing units among the plurality of processing units. Here, the first processing unit 21 and the second processing unit 22 are not used to limit the neural network accelerator to only two processing units.

[0063] In some embodiments, multiple processing units in a neural network accelerator may include a data transfer module, and different processing units may share data through the data transfer module.

[0064] refer to Figure 1 As shown, the first processing unit 21 may include a first data transfer module 211 , and the second processing unit 22 may include a second data transfer module 221 .

[0065] In some embodiments, the first data transfer module 211 in the first processing unit 21 is connected to the second data transfer module 221 in the second processing unit 22 .

[0066] In an embodiment of the present application, the first processing unit 21 sends the first target data to the second data transfer module 221 through the first data transfer module 211, and / or the first processing unit 21 receives the second target data sent by the second data transfer module 221 through the first data transfer module 211.

[0067] In some embodiments, the first target data may be data generated by the first processing unit 21, or data sent by other processing units to the first processing unit 21 via the first data transfer module 211. The second target data may be data generated by the second processing unit 22, or data sent by other processing units to the second processing unit 22 via the second data transfer module 221. The present embodiment does not limit the sources of the first target data and the second target data.

[0068] It can be seen that the processing unit in the embodiment of the present application may include a data transfer module, which forwards the data of the current processing unit to other processing units through the data transfer module, thereby realizing information sharing between different processing units and improving the flexibility of the deep learning algorithm.

[0069] In some embodiments, reference Figure 3 The structure of the neural network accelerator shown Figure 2, the first processing unit may further include a first data storage module 212 and a first data processing module 213;

[0070] The first data transfer module 211 is connected to the first data storage module 212 and the first data processing module 213 respectively;

[0071] A first data storage module 212, used to store data to be processed;

[0072] The first data processing module 213 is used to process the data to be processed and obtain a processing result.

[0073] In an embodiment of the present application, the first data transfer module 211 can be connected to the first data storage module 212 through a first input and output path, the first data transfer module 211 can be connected to the first data processing module 213 through a second input and output path, and the first data transfer module 211 can also be connected to the second data transfer module 221 in the second processing unit 22 through a third input and output path.

[0074] Based on this, the first data transfer module 211 can read the data to be processed stored in the first data storage module 212 through the first input / output path, or write the obtained data to the first data storage module 212. The first data transfer module 211 can read the processing results of the first data processing module 213 through the second input / output path, or write data to the first data processing module 213, so that the first data processing module 213 processes the written data. Furthermore, the first data transfer module 211 can send data to other processing units (i.e., the second processing unit) through the third input / output path, or receive data sent by other processing units.

[0075] It can be understood that the data to be processed stored in the first data storage module 212 and the processing results obtained by the first data processing module 213 can not only be transmitted in the current first processing unit 21, but can also be transmitted to the second processing unit 22 by the first data transfer module 211 for processing.

[0076] In the embodiment of the present application, the second processing unit 22 is similar to the first processing unit 21 and may also include a second data storage module 222 and a second data processing module 223. The second data transfer module 221 in the second processing unit 22 may be connected to the second data storage module 222 and the second data processing module 223, respectively. In this way, the data to be processed and the processing results corresponding to the second processing unit can not only be transmitted within the current second processing unit 22, but can also be transmitted by the second data transfer module 221 to the first processing unit 21 for processing.

[0077] In other words, the first processing unit 21 and the second processing unit 22 can be connected via the first data transfer module 211 and the second data transfer module 221. Each processing unit has its own dedicated storage module, allowing the data of the processing unit to be processed by the current processing unit, improving data operation efficiency and facilitating hardware layout and wiring. In addition, the data of each processing unit can be transmitted to other processing units for processing via the data transfer module, enabling information sharing between different processing units and increasing the flexibility of deep learning algorithms.

[0078] In some embodiments, the first data processing module 213 and the second data processing module 223 may include a convolution processing submodule and a vector processing submodule. The vector processing submodule may perform low-intensity data operations such as pooling, resizing, feature image splicing, and feature image data rearrangement on the data to be processed. The convolution processing submodule may include a multiply-accumulate array that primarily performs convolution operations on the results of the vector processing submodule to obtain processing results corresponding to the data to be processed.

[0079] It should be noted that the first processing unit 21 and the second processing unit 22 are any two adjacent processing units among the multiple processing units in the neural network accelerator provided in the embodiment of the present application. In other words, each processing unit in the neural network accelerator can be connected to its adjacent processing unit through a data transfer module. In this way, the data of multiple processing units can form a data loop through the data transfer module, and the data corresponding to each processing unit can be transmitted to any processing unit for processing. In this way, information sharing between different processing units is achieved, and the flexibility of the deep learning algorithm is improved.

[0080] In some embodiments, the aforementioned multiple processing units may be all processing units in the neural network accelerator, or may be some processing units in the neural network accelerator. In other words, the neural network accelerator may connect all of its processing units through a data transfer module to form a complete data loop. Alternatively, the neural network accelerator may connect some of its processing units through a data transfer module, and connect the remaining processing units of the neural network accelerator through a data transfer module, thereby forming two data loops to reduce data transmission latency.

[0081] In some embodiments, reference Figure 4 The structure of the neural network accelerator shown Figure 3 The first data transfer module 211 in the neural network accelerator provided in the embodiment of the present application may include a forwarding submodule 2111.

[0082] The forwarding submodule 2111 is connected to the first data storage module 212, the first data processing module 213, and the second data transfer module 221 respectively;

[0083] The forwarding submodule 2111 is configured to read the processing result and / or the data to be processed; and send the processing result and / or the data to be processed as the first target data to the second data transfer module 221 .

[0084] In the embodiment of the present application, the forwarding submodule 2111 can be connected to the first data storage module 212 via the first input / output path. In this way, the forwarding submodule 2111 can read the data to be processed from the first data storage module 212 using the input path of the first input / output path, and write the data to the first data storage module 212 using the output path of the first input / output path.

[0085] The forwarding submodule 2111 can be connected to the first data processing module 213 through the second input and output path. In this way, the forwarding submodule 2111 can use the output path in the second input and output path to write the data to be processed to the first data processing module 213, and use the input path in the second input and output path to read the processing results corresponding to the data to be processed from the first data processing module 213.

[0086] The forwarding submodule 2111 can be connected to the second data transfer module 221 in the second processing unit 22 via the third input / output path. In this way, the forwarding submodule 2111 can use the output path in the third input / output path to send the first target data to the second data transfer module 221, and use the input path in the first input / output path to receive the second target data sent by the second data transfer module 221.

[0087] In some embodiments, the second data transfer module 221 may also include a forwarding submodule 2211 . The third input / output path is specifically connected to the forwarding submodule 2211 in the second data transfer module 221 .

[0088] In an embodiment of the present application, the forwarding submodule 2111 can be controlled to connect at least part of the first input and output paths, the second input and output paths, and the third input and output paths, so that the data to be processed and / or the processing results in the first processing unit 21 can not only be transmitted in the first processing unit 21, but also can be transmitted to the second processing unit 22 for processing.

[0089] In some embodiments, a central processing unit (e.g., a CPU) may control the opening and closing of multiple input and output paths in the forwarding submodule 2111 according to a specific deep learning algorithm to implement different functions of the deep learning algorithm.

[0090] In one example, the forwarding submodule 2111 can connect the input path of the first input / output path and the output path of the third input / output path under the control of the CPU. Thus, the forwarding submodule 2111 can read the data to be processed from the first data storage module 212 via the input path of the first input / output path, and send the read data to be processed as the first target data to the second data transfer module 221 of the second processing unit 22 via the output path of the third input / output path.

[0091] In another example, the forwarding submodule 2111 can connect the input path of the second input / output path and the output path of the third input / output path under the control of the CPU. In this way, the forwarding submodule 2111 can read the processing result of the first data processing module 213 through the input path of the second input / output path and send the read processing result as the first target data to the second data transfer module 221 of the second processing unit 22 through the output path of the third input / output path.

[0092] In another example, the forwarding submodule 2111 can, under CPU control, connect the input path of the first input / output path, the input path of the second input / output path, and the output path of the third input / output path. In this way, the forwarding submodule 2111 can read the unprocessed data from the first data storage module 212 via the input path of the first input / output path, and read the processing results of the first data processing module 213 via the input path of the second input / output path. Furthermore, the forwarding submodule 2111 can send the read unprocessed data and processing results as first target data to the second data transfer module 221 of the second processing unit 22 via the output path of the third input / output path.

[0093] It can be seen that the processing unit in the neural network accelerator provided in the embodiment of the present application can forward the data of the current processing unit to other processing units through the data transfer module, thereby realizing information sharing between different processing units and improving the flexibility of the deep learning algorithm.

[0094] In some embodiments, reference Figure 5 The structure of the neural network accelerator shown Figure 4 , the first data transfer module 211 may include a forwarding submodule 2111 and an operation submodule 2112;

[0095] The operation submodule 2112 may be connected to the first data storage module 212 , the first data processing module 213 , and the forwarding submodule 2111 , respectively; the forwarding submodule 2111 may also be connected to the second data transfer module 221 .

[0096] That is to say, in addition to setting up a forwarding submodule 2111 to realize data sharing between different processing units, the first data transfer module 211 in the embodiment of the present application can also set up an operation submodule 2112 to realize data operation between different processing units.

[0097] In some embodiments, the operation submodule 2112 can be connected to the first data storage module 212 through a fourth input-output path, use the input path in the fourth input-output path to read the data to be processed from the first data storage module 212, and use the output path in the fourth input-output path to write data to the first data storage module 212.

[0098] The operator module 2112 can be connected to the first data processing module 213 through the fifth input and output path, read the processing results from the first data processing module 213 using the input path of the fifth input and output path, and write data to the first data processing module 213 using the output path of the fifth input and output path.

[0099] The operation submodule 2112 can be connected to the forwarding submodule 2111 through the sixth input-output path, use the input path of the sixth input-output path to read the second target data forwarded by the second processing unit from the forwarding submodule 2111, and use the output path of the sixth input-output path to transmit the operation result to the forwarding submodule 2111.

[0100] Correspondingly, the forwarding submodule 2111 can be connected to the second data transfer module 221 in the second processing unit 22 through the seventh input-output path, read the second target data from the second data transfer module 221 using the input path of the seventh input-output path, and forward the calculation result sent by the calculation submodule 2112 to the second data transfer module 221 using the output path of the seventh input-output path.

[0101] In some embodiments, second data transfer module 221 is similar to first data transfer module 211 and may also include a forwarding submodule and a computation submodule. The forwarding submodule forwards data, while the computation submodule performs computations on different data. The seventh input / output path is specifically connected to the forwarding submodule in second data transfer module 221.

[0102] In some embodiments, the operation submodule 2112 can perform operation processing on the processing results and / or the data to be processed to obtain a first operation result; correspondingly, the forwarding submodule 2111 can send the first operation result as the first target data to the second data transfer module 221.

[0103] That is, the first data transfer module 211 can perform computational processing on the data in the current first processing unit and forward the processing results to other processing units for further processing.

[0104] Specifically, the operation submodule 2112 can be controlled to connect at least part of the input and output paths from the fourth input and output path to the seventh input and output path, so that the data to be processed and / or processing results in the first processing unit 21, and the second target data sent by the second processing unit 22 can be calculated in the operation submodule 2112.

[0105] In one example, under CPU control, the operation submodule 2112 can connect the input path of the fourth input / output path, and the output paths corresponding to the sixth and seventh input / output paths. In this way, the operation submodule 2112 can read the unprocessed data from the first data storage module 212 via the input path of the fourth input / output path, and perform operations on the read unprocessed data to obtain a first operation result. Furthermore, the operation submodule 2112 transmits the obtained first operation result to the forwarding submodule 2111 via the output path of the sixth input / output path. In this way, the forwarding submodule 2111 can forward the first operation result as the first target data to the second data transfer module 221 via the output path of the seventh input / output path.

[0106] In another example, under CPU control, the operation submodule 2112 can connect the input path of the fifth input / output path, and the output paths corresponding to the sixth and seventh input / output paths. In this way, the operation submodule 2112 can read the processing result from the first data processing module 213 via the input path of the fifth input / output path, and perform operation processing on the read processing result to obtain a first operation result. Furthermore, the operation submodule 2112 sends the obtained first operation result to the forwarding submodule 2111 via the output path of the sixth input / output path. In this way, the forwarding submodule 2111 can forward the first operation result as the first target data to the second data transfer module 221 via the output path of the seventh input / output path.

[0107] In another example, under CPU control, the operation submodule 2112 can connect the corresponding input paths of the fourth and fifth input / output paths, as well as the corresponding output paths of the sixth and seventh input / output paths. In this way, the operation submodule 2112 can read the unprocessed data stored in the first data storage module 212 via the input path of the fourth input / output path, read the processing result from the first data processing module 213 via the input path of the fifth input / output path, and perform operation processing on the read unprocessed data and the processing result to obtain a first operation result. Furthermore, the operation submodule 2112 transmits the obtained first operation result to the forwarding submodule 2111 via the output path of the sixth input / output path. In this way, the forwarding submodule 2111 can forward the first operation result as the first target data to the second data transfer module 221 via the output path of the seventh input / output path.

[0108] In some embodiments, the forwarding submodule 2111 is also used to receive the second target data sent by the second data transfer module 221; correspondingly, the operation submodule 2112 is also used to perform operation processing on at least one of the second target data, the data to be processed, and the processing results to obtain a second operation result.

[0109] That is, the first data transfer module 211 can receive the second target data sent by the second processing unit, and use the operation processing submodule 2112 to perform operation processing on the second target data and the data in the current first processing unit to obtain a second operation result.

[0110] In one example, under CPU control, the operation submodule 2112 can connect the input paths corresponding to the sixth and seventh input / output paths. Thus, the forwarding submodule 2111 can read the second target data sent by the second data transfer module 221 through the input path of the seventh input / output path, and the operation submodule 2112 can read the second target data through the input path of the sixth input / output path and perform an operation on the second target data to obtain a second operation result.

[0111] In another example, the operation submodule 2112 can connect the input paths corresponding to the fourth input / output path, the sixth input / output path, and the seventh input / output path under CPU control. In this way, the forwarding submodule 2111 can read the second target data sent by the second data transfer module 221 through the input path of the seventh input / output path. The operation submodule 2112 can read the second target data through the input path of the sixth input / output path, read the data to be processed from the first data storage module 212 through the input path of the fourth input / output path, and perform operation processing on the second target data and the data to be processed to obtain a second operation result.

[0112] In another example, the operation submodule 2112 can connect the input paths corresponding to the fifth input / output path, the sixth input / output path, and the seventh input / output path under CPU control. In this way, the forwarding submodule 2111 can read the second target data sent by the second data transfer module 221 through the input path of the seventh input / output path. The operation submodule 2112 can read the second target data through the input path of the sixth input / output path, read the processing result from the first data processing module 213 through the input path of the fifth input / output path, and perform operation processing on the second target data and the processing result to obtain a second operation result.

[0113] It can be understood that the neural network accelerator provided in the embodiment of the present application can add a data transfer module to each processing unit, so that data can be shared between the processing units. At the same time, calculation operations can be performed on the data of different processing units within the data transfer module, further improving the flexibility of algorithm implementation.

[0114] In some embodiments, after the operation submodule 2112 obtains the second operation result, the second operation result can be written into the first data storage module 212 through the output path of the fourth input-output path, or the second operation result can be transmitted to the first data processing module 213 for processing through the output path of the fifth input-output path, and the second operation result can also be sent to the forwarding submodule 2111 through the output path of the sixth input-output path, so that the forwarding submodule 2111 forwards the obtained second operation result to other processing units. The embodiments of the present application do not limit this.

[0115] refer to Figure 6 The neural network accelerator structure shown Figure 5 The neural network accelerator provided in this embodiment of the present application may further include a third processing unit 23.

[0116] Here, the third processing unit 23 can be a processing unit among multiple processing units in the neural network accelerator, which is different from the first processing unit 21 and the second processing unit 22.

[0117] In some embodiments, the first data transfer module 211 in the first processing unit 21 can be connected to the third data transfer module 231 in the third processing unit 23. Specifically, the forwarding submodule 2111 in the first data transfer module 211 can be connected to the third data transfer module 231 via an eighth input / output path. The forwarding submodule 2111 can read data from the third data transfer module 231 using the input path of the eighth input / output path and send data to the third data transfer module using the output path of the eighth input / output path.

[0118] In some embodiments, the third processing unit 23 may be located adjacent to the first processing unit 21, while the third processing unit 23 may not be located adjacent to the second processing unit 22. In other words, the third processing unit 23, the first processing unit 21, and the second processing unit 22 are arranged sequentially, and the three processing units may be connected via their respective data transfer modules to form a data loop.

[0119] Based on the above structure, in some embodiments, the forwarding submodule 2111 is further configured to send the second operation result to the third data transfer module 231 .

[0120] Specifically, after obtaining the second operation result, the operation submodule 2112 in the first data transfer module 211 can send the second operation result to the forwarding submodule 2111 via the output path of the sixth input / output path. In this way, the forwarding submodule 2111 can send the obtained second operation result to the third data transfer module 231 via the output path of the eighth input / output path.

[0121] For example, a deep learning algorithm requires that the data A of channel 0 be bit-inverted and added to the data B of channel 1, and the result is stored in the storage module corresponding to channel 2, that is, If channel 0 corresponds to processing unit 0, channel 1 corresponds to processing unit 1, and channel 2 corresponds to processing unit 2, refer to Figure 7 As shown in the data transmission diagram, the operation submodule in the processing unit 0 can read the data A from the data storage module 0 and perform bitwise inversion operation to obtain the data Then, the forwarding submodule of processing unit 0 sends the data Forwards to the forwarding submodule in the processing unit 1, and the operation submodule of the processing unit 1 can read the data obtained by the current forwarding submodule And the data B stored in the current processing unit 1, so that the received data Add it to data B and get data Furthermore, the forwarding submodule in the processing unit 1 can Forwards the received data to the forwarding submodule of the processing unit 2. The forwarding submodule can Stored in the data storage module of the current processing unit 2. In this way, it can be achieved operation.

[0122] Thus, the neural network accelerator provided in the embodiments of the present application adds a data transfer module to each processing unit, enabling data sharing among the processing units. This solves the problem of data sharing among distributed storage units in neural network accelerators used for deep learning computations. Furthermore, the data transfer module is internally provided with a computational submodule that can perform operations on data from different processing units, expanding the hardware architecture's support for algorithms.

[0123] In some embodiments, the operation submodule 2112 may include at least one unary operation circuit and / or at least one binary operation circuit. The operation submodule 2112 may utilize the unary operation circuit to implement unary operations on data, such as bitwise inversion, exponentiation, and square root operations. The operation submodule 2112 may utilize the binary operation circuit to implement binary operations such as addition, subtraction, multiplication, and division between two data items. For example, the operation submodule 2112 may perform an addition operation on the second target data from the second processing unit 22 and the data to be processed in the current first processing unit 21, and store the result of the addition operation in the first data storage module 212 of the first processing unit 21.

[0124] It should be noted that the operator module 2112 may include only a unary operation circuit, only a binary operation circuit, or both a unary operation circuit and a binary operation circuit, and this is not limited in the present embodiment. In addition, this embodiment does not limit the number of unary operation circuits and binary operation circuits.

[0125] In some embodiments, when the operation submodule 2112 includes multiple operation circuits, the central processing unit can control the operation circuits in the operation submodule 2112 to work through an enable signal according to the specific deep learning algorithm requirements, and control the operation submodule 2112 to implement different operation operations.

[0126] In some embodiments, the operation type of the at least one unary operation circuit and / or the at least one binary operation circuit can be determined based on configuration information. That is, the central processing unit can receive the user's configuration information, and the central processing unit can configure the operation type implemented by the operation circuit in the operation submodule 2112 based on the received configuration information. For example, the binary operation circuit can be configured to implement a subtraction operation or an addition operation through the configuration information. In this way, the operation function of the operation submodule 2112 can be expanded, thereby improving the flexibility of the neural network accelerator.

[0127] In some embodiments, the first target data sent by the first processing unit 21 to the second data transfer module 221 of the second processing unit 22 through the first data transfer module 211 may include at least one of the following:

[0128] The data to be processed is stored in the first data storage module;

[0129] a processing result obtained by the first data processing module;

[0130] The third target data is sent by the third processing unit.

[0131] Correspondingly, in some embodiments, the second target data sent by the second processing unit 22 to the first data transfer module 211 of the first processing unit 21 through the second data transfer module 221 may include at least one of the following:

[0132] The data to be processed is stored in the second data storage module of the second processing unit;

[0133] a processing result obtained by processing by the second data processing module of the second processing unit;

[0134] The fourth target data is sent by the fourth processing unit.

[0135] refer to Figure 8 The neural network accelerator structure shown Figure 6 , the fourth processing unit 24 can be a processing unit adjacent to the second processing unit 22. The fourth processing unit 24 is similar to other processing units and can include a fourth data transfer module 241, a fourth data storage module 242 and a fourth data processing module 243. Among them, the fourth data transfer module 241 can be connected to the second data transfer module 221 of the second processing unit 22. Based on this, the first processing unit 21, the second processing unit 22, the third processing unit 23 and the fourth processing unit 24 are connected through the data transfer module to form a data loop. In this way, each processing unit can obtain the data of any processing unit in the data loop, solving the problem that data cannot be shared between different processing units. In addition, when the neural network accelerator provided in the embodiment of the present application needs to expand the processing unit, only a data transfer module can be added to the expanded processing unit to share data with other processing units, without changing the design of the existing data loop, thereby improving the flexibility of hardware expansion.

[0136] The following further describes the neural network accelerator provided in the embodiments of the present application in combination with specific application scenarios.

[0137] refer to Figure 9 The structure of the neural network accelerator shown Figure 7 The neural network accelerator provided in the embodiment of the present application may include N processing units, each of which may include a convolution processing module, a vector processing module, a data transfer module and a data storage module.

[0138] The data transfer module in each processing unit is provided with four pairs of input and output paths. The first pair of input and output paths can be used to read data from the data storage module in the current processing unit and write data to the data storage module in the current processing unit. The second pair of input and output paths can be used to read data from the data processing module in the current processing unit or write data to the data processing module in the current processing unit. The third and fourth pairs of input and output paths are paths for the data transfer module to forward data to two adjacent processing units. In this way, the data in each processing unit is connected in a ring through the data transfer module in each processing unit, forming a data loop. The data of each processing unit can not only be used by the data processing module of the processing unit itself, but can also be transmitted to other processing units for processing.

[0139] In other words, each processing unit can occupy a separate data storage module. Utilizing a distributed storage and distributed computing architecture, data from each processing unit can be processed in parallel, improving data processing efficiency. The overall hardware layout is neat, facilitating later layout and routing. Furthermore, each processing unit is connected via a data transfer module, enabling information sharing between different processing units and increasing the flexibility of deep learning algorithms.

[0140] based on Figure 9 As shown in the structural diagram of the neural network accelerator, when the deep learning algorithm needs to shuffle the data of channel 0 to channel 4 (corresponding to processing unit 0 to processing unit 4), processing unit 0 to processing unit 3 can read the data in their respective data storage modules through their respective data transfer modules, and forward the data to the data transfer module corresponding to processing unit 4 through the data transfer module. In this way, processing unit 4 can combine the data of processing unit 0 to processing unit 3 based on the data of the current processing unit 4 and write it to the data storage module of processing unit 4. In this way, reference Figure 10 As shown in the shuffle operation diagram, the central processing unit can perform a shuffle operation on the data of each channel, that is, rearrange the data of each channel.

[0141] In summary, in the neural network accelerator provided by the embodiments of the present application, each processing unit has its own dedicated data storage module, which improves data processing efficiency and facilitates layout and wiring. In addition, each processing unit is connected through a data transfer module to share data, which improves the flexibility of deep learning algorithms.

[0142] An embodiment of the present application also provides a data processing device, which may include the neural network accelerator provided in the above embodiment.

[0143] The data processing device can realize the functions of the neural network accelerator time in the above-mentioned various structures in the embodiments of the present application. For the sake of brevity, it will not be repeated here.

[0144] In the embodiment of the present application, the data processing device may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. The embodiment of the present application is not limited to this.

[0145] The present application also provides a neural network acceleration method, which can be applied to the neural network accelerator provided in the above embodiment. Figure 11 The neural network acceleration method shown in FIG. 1 may include the following steps:

[0146] Step 1110: Control the first processing unit to send the first target data to the second data transfer module through the first data transfer module; so that the second processing unit processes the first target data; and / or control the first processing unit to receive the second target data sent by the second data transfer module through the first data transfer module and process the second target data.

[0147] It is understood that in the neural network acceleration method provided in the embodiments of the present application, the first processing unit can receive the second target data from the second processing unit through the first data transfer module for processing, and at the same time, the second processing unit can also receive the first target data from the first processing unit through the second data transfer module for processing. In this way, information sharing is achieved between different processing units, improving the flexibility of deep learning algorithms.

[0148] In some embodiments, the first processing unit in the neural network accelerator further includes a first data storage module and a first data processing module;

[0149] The first data transfer module is connected to the first data processing module and the first data storage module respectively;

[0150] The first data storage module is used to store data to be processed;

[0151] The first data processing module is used to process the data to be processed to obtain a processing result.

[0152] In some embodiments, the first data transfer module includes a forwarding submodule;

[0153] The forwarding submodule is connected to the first data storage module, the first data processing module, and the second data transfer module respectively;

[0154] Controlling the first processing unit to send the first target data to the second data transfer module via the first data transfer module in step 1110 can be achieved by the following steps:

[0155] Control the forwarding submodule in the first data transfer module to read the processing result and / or the data to be processed; and send the processing result and / or the data to be processed as the first target data to the second data transfer module.

[0156] In some embodiments, the first data transfer module includes an operation submodule and a forwarding submodule;

[0157] The operation submodule is connected to the first data storage module, the first data processing module, and the forwarding submodule respectively;

[0158] The forwarding submodule is also connected to the second data transfer module;

[0159] Controlling the first processing unit to send the first target data to the second data transfer module via the first data transfer module in step 1110 can be achieved by the following steps:

[0160] Controlling the operation submodule in the first data transfer module to perform operation processing on the processing result and / or the data to be processed to obtain a first operation result;

[0161] The forwarding submodule in the first data transfer module is controlled to send the first operation result as the first target data to the second data transfer module.

[0162] In some embodiments, controlling the first processing unit to receive the second target data sent by the second data transfer module through the first data transfer module and processing the second target data in step 1110 can be achieved by the following steps:

[0163] Controlling the forwarding submodule to receive the second target data sent by the second data transfer module;

[0164] Controlling the operation submodule to perform operation processing on at least one of the second target data, the data to be processed, and the processing result to obtain a second operation result;

[0165] The first data processing module processes the second operation result.

[0166] In some embodiments, the neural network accelerator further includes a third processing unit; wherein a third data transfer module of the third processing unit is connected to the forwarding submodule;

[0167] The method further comprises:

[0168] Control the forwarding submodule in the first transfer module to send the second operation result to the third data transfer module.

[0169] In some embodiments, the operation submodule includes at least one unary operation circuit and / or at least one binary operation circuit;

[0170] The operation type of the at least one unary operation circuit and / or the at least one binary operation circuit is determined based on the configuration information.

[0171] In some embodiments, the second target data includes at least one of the following:

[0172] data to be processed stored in the second data storage module of the second processing unit;

[0173] a processing result obtained by processing by the second data processing module of the second processing unit;

[0174] The fourth target data is sent by the fourth processing unit; the fourth data transfer module of the fourth processing unit is connected to the second data transfer module.

[0175] In some embodiments, the first processing unit and the second processing unit are any two adjacent processing units among the multiple processing units included in the neural network accelerator.

[0176] It should be understood that the neural network accelerator in the embodiment of the present application is the same as the neural network accelerator in the above embodiment. For the sake of brevity, it will not be repeated here.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or at least two units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0180] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, ROM, RAM, disks or optical disks, etc. Various media that can store program codes.

[0181] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0182] It should be noted that the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0183] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A neural network accelerator, characterized in that: include: a first processing unit and a second processing unit; The first data transfer module in the first processing unit is connected to the second data transfer module in the second processing unit; The first processing unit sends the first target data to the second data transfer module through the first data transfer module, and / or the first processing unit receives the second target data sent by the second data transfer module through the first data transfer module; The first processing unit further includes a first data storage module and a first data processing module; The first data transfer module is connected to the first data processing module and the first data storage module respectively; The first data storage module is used to store data to be processed; The first data processing module is used to process the data to be processed to obtain a processing result; The second processing unit further includes a second data storage module and a second data processing module; wherein the second data transfer module is connected to the second data storage module and the second data processing module respectively; The first data transfer module includes an operation submodule and a forwarding submodule; The operation submodule includes at least one unary operation circuit and / or at least one binary operation circuit; The operation submodule is connected to the first data storage module, the first data processing module, and the forwarding submodule respectively; The forwarding submodule is also connected to the second data transfer module; The operation submodule is used to perform operation processing on the processing result and / or the data to be processed to obtain a first operation result; The forwarding submodule is configured to send the first operation result as the first target data to the second data transfer module.

2. The neural network accelerator according to claim 1, characterized in that The first data transfer module includes a forwarding submodule; The forwarding submodule is connected to the first data storage module, the first data processing module, and the second data transfer module respectively; The forwarding submodule is configured to read the processing result and / or the data to be processed; The processing result and / or the data to be processed are sent to the second data transfer module as the first target data.

3. The neural network accelerator according to claim 1, wherein: The forwarding submodule is further configured to receive the second target data sent by the second data transfer module; The operation submodule is further configured to perform operation processing on at least one of the second target data, the data to be processed, and the processing result to obtain a second operation result.

4. The neural network accelerator according to claim 3, characterized in that The neural network accelerator further includes a third processing unit; The third data transfer module of the third processing unit is connected to the forwarding submodule; The forwarding submodule is further configured to send the second operation result to the third data transfer module.

5. The neural network accelerator according to any one of claims 1, 3 and 4, characterized in that: The operation type of the at least one unary operation circuit and / or the at least one binary operation circuit is determined based on the configuration information.

6. The neural network accelerator according to any one of claims 1 to 3, wherein: The second target data includes at least one of the following: data to be processed stored in the second data storage module of the second processing unit; a processing result obtained by processing by the second data processing module of the second processing unit; The fourth target data is sent by the fourth processing unit; the fourth data transfer module of the fourth processing unit is connected to the second data transfer module of the second processing unit.

7. The neural network accelerator according to claim 1, wherein: The first processing unit and the second processing unit are any two adjacent processing units among the multiple processing units included in the neural network accelerator.

8. A data processing device comprising the neural network accelerator according to any one of claims 1 to 7.

9. A neural network acceleration method, characterized in that: Applied to a neural network accelerator, the neural network accelerator includes a first processing unit and a second processing unit; a first data transfer module in the first processing unit is connected to a second data transfer module in the second processing unit; The method comprises: controlling the first processing unit to send first target data to the second data transfer module through the first data transfer module, so that the second processing unit processes the first target data; and / or, controlling the first processing unit to receive, through the first data transfer module, second target data sent by the second data transfer module, and to process the second target data; The first processing unit further includes a first data storage module and a first data processing module; the first data transfer module is connected to the first data processing module and the first data storage module respectively; the first data storage module is used to store data to be processed; the first data processing module is used to process the data to be processed and obtain a processing result; the second processing unit further includes a second data storage module and a second data processing module; the second data transfer module is connected to the second data storage module and the second data processing module respectively; The controlling the first processing unit to send the first target data to the second data transfer module through the first data transfer module includes: Controlling the operation submodule in the first data transfer module to perform operation processing on the processing result and / or the data to be processed to obtain a first operation result; Control the forwarding submodule in the first data transfer module to send the first operation result as the first target data to the second data transfer module; wherein, the first data transfer module includes an operation submodule and a forwarding submodule; the operation submodule includes at least one unary operation circuit, and / or at least one binary operation circuit; the operation submodule is respectively connected to the first data storage module, the first data processing module, and the forwarding submodule; the forwarding submodule is also connected to the second data transfer module.

Images