Accelerator Device
The accelerator device optimizes neural network processing by using multiple specialized circuits with tailored configurations and timing management to enhance efficiency and reduce memory access conflicts, addressing inefficiencies in existing hardware configurations.
Patent Information
- Application Number
- JP2022205259
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-12-22
- Publication Date
- 2026-02-25
- Estimated Expiration
- 2042-12-22
AI Technical Summary
In neural networks, the efficiency of arithmetic processing decreases due to mismatches between hardware configurations and the types of operations performed, leading to suboptimal performance when multiple computational processes with different operations are combined.
An accelerator device with multiple accelerator circuits, each equipped with specific arithmetic units, caches, and storage units, uses allocation information to determine the optimal timing for starting operations in each circuit, ensuring suitable hardware configurations for each operation.
This approach enhances processing efficiency by executing operations in circuits tailored for each type of operation, reducing the likelihood of memory access bottlenecks and maintaining high performance across diverse neural network tasks.
Smart Images
Figure 0007820287000001 
Figure 0007820287000002 
Figure 0007820287000003
Abstract
Description
[Technical Field]
[0001] The present disclosure relates to an accelerator device. [Background technology]
[0002] In arithmetic processing using a multi-layer neural network, an accelerator configured with hardware suitable for arithmetic processing in the neural network is sometimes used (Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] US Patent Application Publication No. 2019 / 0205737A1 Summary of the Invention [Problem to be solved by the invention]
[0004] However, when multiple computational processes with different types of operations are performed in a neural network, the appropriate hardware configuration may differ depending on the type of operation, and processing efficiency may decrease depending on the combination of the type of operation and the hardware configuration. [Means for solving the problem]
[0005] The present disclosure can be realized in the following forms.
[0006] According to one embodiment of the present disclosure, there is provided an accelerator device (100) for use in arithmetic processing using a neural network, the accelerator device (100) comprising: a plurality of accelerator circuits (ACC1, ACC2, ACC3, ACC4, ACC5), each having an arithmetic unit (14, 24, 34, 44, 54), a cache unit (12, 22, 32, 42, 52), and a storage unit (11, 21, 31, 41, 51) realized by a semiconductor memory device (111); a register (120) that stores allocation information indicating an assigned circuit, which is an accelerator circuit among the plurality of accelerator circuits, each configured to execute a predetermined type of arithmetic processing, and which is set in correspondence with each of a plurality of arithmetic processing operations executed by the neural network; and a scheduler (132) that uses the allocation information to determine a timing for starting each of the plurality of arithmetic processing operations in each of the accelerator circuits.
[0007] According to this form of accelerator device, the accelerator device has a plurality of accelerator circuits, and uses allocation information indicating the assigned circuits, which are accelerator circuits set up so as to correspond one by one to each arithmetic operation, to determine the timing for starting arithmetic operations in each accelerator device. Therefore, arithmetic operations can be executed in accelerator devices having a configuration suitable for each arithmetic operation, and a decrease in the efficiency of the arithmetic operations can be suppressed. [Brief explanation of the drawings]
[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of an accelerator device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing a schematic configuration of an arithmetic circuit according to the present embodiment. [Figure 3] FIG. 2 is an explanatory diagram illustrating an example of the configuration of a neural network realized in the accelerator device of the present embodiment. [Figure 4]FIG. 2 is an explanatory diagram illustrating an example of the configuration of a neural network realized in the accelerator device of the present embodiment. [Figure 5] 10 is a flowchart illustrating a procedure for a scheduling process according to the present embodiment. [Figure 6] FIG. 10 is an explanatory diagram showing an example of a calculation processing schedule determined in step S30 of the scheduling process. DETAILED DESCRIPTION OF THE INVENTION
[0009] A. First embodiment: A-1.Device configuration: The accelerator device 100 of this embodiment executes arithmetic processing using a neural network. In this embodiment, the accelerator device 100 performs arithmetic processing on image data captured by a camera mounted on a vehicle, outputting feature quantities for identifying pedestrians, structures, and the like contained in the image data. As shown in FIG. 1, the accelerator device 100 includes an arithmetic circuit 110, a register 120, and an arithmetic control unit 130. Note that the accelerator device 100 may execute arithmetic processing on voice data or text data in addition to image data.
[0010] As shown in FIG. 2, the arithmetic circuit 110 includes a first accelerator circuit ACC1, a second accelerator circuit ACC2, a third accelerator circuit ACC3, a fourth accelerator circuit ACC4, and a fifth accelerator circuit ACC5. The first accelerator circuit ACC1 includes a first memory unit 11, a first cache unit 12, a first buffer unit 13, and a first product-sum operation unit 14. The second accelerator circuit ACC2 includes a second memory unit 21, a second cache unit 22, a second buffer unit 23, and a second product-sum operation unit 24. The third accelerator circuit ACC3 includes a third memory unit 31, a third cache unit 32, a third buffer unit 33, and a pooling operation unit 34. The fourth accelerator circuit ACC4 includes a fourth memory unit 41, a fourth cache unit 42, a fourth buffer unit 43, and an activation operation unit 44. The fifth accelerator circuit ACC5 includes a fifth storage unit 51, a fifth cache unit 52, a fifth buffer unit 53, and a vector operation unit .
[0011] The first memory unit 11, the second memory unit 21, the third memory unit 31, the fourth memory unit 41, and the fifth memory unit 51 (hereinafter collectively referred to as "memories") are realized as different memory areas of the memory area of the semiconductor memory device 111. In this embodiment, the semiconductor memory device 111 is configured as a DRAM. Each memory unit stores image data input and output in arithmetic processing. In addition, the first memory unit 11 and the second memory unit 21 store filter coefficients and weighting coefficients used in product-sum operations. The first memory unit 11 and the second memory unit 21 have different memory capacities, with the first memory unit 11 having a larger memory capacity than the second memory unit 21.
[0012] The first cache unit 12, the second cache unit 22, the third cache unit 32, the fourth cache unit 42, and the fifth cache unit 52 (hereinafter collectively referred to as "cache units") are realized as different storage areas within the storage area of the cache device 112. Each cache unit stores frequently used data and the most recently read data among the data stored in each storage unit in each of the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5. This reduces the frequency of access to storage units that require a long time for data access, thereby preventing a decrease in data access efficiency. The first cache unit 12 and the second cache unit 22 have different storage capacities, with the first cache unit 12 having a larger storage capacity than the second cache unit 22.
[0013] In this embodiment, the first product-sum calculation unit 14 and the second product-sum calculation unit 24 perform convolution calculations on an input image using filter data and weighting coefficients. In this embodiment, the first product-sum calculation unit 14 and the second product-sum calculation unit 24 have different product-sum calculation capabilities, and the second product-sum calculation unit 24 performs more product-sum calculations per unit time than the first product-sum calculation unit 14. In addition, the second product-sum calculation unit 24 is configured to be able to perform multiple calculation processes in parallel.
[0014] The pooling calculation unit 34 performs pooling calculation. In this embodiment, the pooling calculation unit 34 performs max pooling or average pooling in the pooling calculation.
[0015] The activation calculation unit 44 performs activation calculation. In this embodiment, the activation calculation unit 44 uses a softmax function or a ReLu function as an activation function in the activation calculation.
[0016] The vector calculation unit 54 performs vector calculations. Hereinafter, the first product-sum calculation unit 14, the second product-sum calculation unit 24, the pooling calculation unit 34, the activation calculation unit 44, and the vector calculation unit 54 will be collectively referred to as the “calculation unit.”
[0017] The first buffer unit 13, the second buffer unit 23, the third buffer unit 33, the fourth buffer unit 43, and the fifth buffer unit 53 (hereinafter collectively referred to as "buffer units") are provided between each cache unit and each arithmetic unit in each of the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5, and temporarily store data when data is transferred between each cache unit and each arithmetic unit. This makes it possible to mitigate differences in data processing speed between each cache unit and each arithmetic unit. The storage capacity of each buffer unit can be determined arbitrarily depending on the magnitude of the difference in data processing speed in each of the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5.
[0018] In this embodiment, the arithmetic unit, buffer unit, and cache unit are configured on the same chip, which is different from the chip on which the storage unit is configured.
[0019] 1 stores information indicating the processing capacity of each of the first to fifth accelerator circuits ACC1 to ACC5 (hereinafter also referred to as "hardware feature information"), and information indicating the type of arithmetic processing and processing load in each layer constituting a neural network (described later) (hereinafter also referred to as "layer feature information"). In this embodiment, the register 120 stores, as the hardware feature information, the type and processing capacity of arithmetic processing by each arithmetic unit in the first to fifth accelerator circuits ACC1 to ACC5, the storage capacity of each memory unit, each cache unit, and each buffer unit, the data transfer rate between each arithmetic unit and each cache unit, and the data transfer rate between each cache unit and each memory unit.
[0020] The layer feature information will be described using the neural network shown in FIGS. 3 and 4 executed by the accelerator device 100 of this embodiment as an example. The neural network shown in FIGS. 3 and 4 is realized by the accelerator device 100 executing a program pre-stored in an external storage device (not shown). The neural network shown in FIG. 3 includes a convolutional layer L11, a pooling layer L12, a residual processing layer L13, a pooling layer L14, a fully connected layer L15, and an activation function layer L16, and the accelerator device 100 executes processing in each layer in this order. The neural network shown in FIG. 4 includes a convolutional layer L21, a residual processing layer L22, a convolutional layer L23, a pooling layer L24, a convolutional layer L25, a fully connected layer L26, and an activation function layer L27, and the accelerator device 100 executes processing in each layer in this order. In addition, in FIG. 3 and FIG. 4, layers having the same name indicate that the same type of calculation processing is executed on the layers.
[0021] The types of computational processing and processing load in each layer are explained below. In the convolution layer, a product-sum operation is performed. In the pooling layer, a pooling operation is performed. In the residual processing layer, residual processing by residual connection is performed in addition to the product-sum operation, so the computational load is higher than in the convolution layer and a larger storage capacity is required. In the fully connected layer, a vector operation is performed to calculate features for all data that make up the image data. In the activation function layer, an activation operation is performed.
[0022] The register 120 shown in FIG. 1 stores information about allocation destination circuits (hereinafter also referred to as "allocation information"), which will be described later, in addition to the hardware characteristic information and layer characteristic information described above.
[0023] The arithmetic control unit 130 includes a layer allocation unit 131 and a scheduler 132 as functional units. The layer allocation unit 131 uses the hardware characteristic information and layer characteristic information stored in the register 120 to set an allocation circuit for each layer. The "allocation circuit" refers to an accelerator circuit among the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5 that is determined to execute processing in each layer and is set in association with each layer. The specific relationship between each layer and the allocation circuit will be explained in the scheduling process described later. The scheduler 132 uses the allocation information to determine the timing to start processing in each layer in each of the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5.
[0024] A-2. Scheduling process: 5, the accelerator device 100 executes a scheduling process that determines the timing to start processing in each layer in each of the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5, using the hardware feature information and layer feature information stored in the register 120. The scheduling process is executed when the accelerator device 100 is started and a program that realizes a neural network is started.
[0025] In step S10, the layer allocation unit 131 acquires the hardware characteristic information and the layer characteristic information stored in the register 120.
[0026] In step S20, the layer allocation unit 131 determines an allocation destination circuit for each layer using the acquired hardware feature information and layer feature information, and records the allocation information in the register 120. Specifically, since the processing in the residual processing layer requires a larger memory capacity than the convolutional layer as described above, it is preferably allocated to the second accelerator circuit ACC2 having the second memory unit 21 and the second cache unit 22 with larger memory capacity. The processing in the convolutional layer is preferably allocated to the first accelerator circuit ACC1 having the first product-accumulation operation unit 14 capable of product-accumulation operation. The processing in the pooling layer is preferably allocated to the third accelerator circuit ACC3 having the pooling operation unit 34 capable of pooling operation. The processing in the activation function layer is preferably allocated to the fourth accelerator circuit ACC4 having the activation operation unit 44 capable of activation operation. The processing in the fully connected layer is preferably allocated to the fifth accelerator circuit ACC5 having the vector operation unit 54 capable of vector operation.
[0027] In step S30, the scheduler 132 uses the allocation information to determine the timing at which processing in each layer is started in each of the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5. An example of the arithmetic processing schedule determined in this step is shown in FIG. 6. FIG. 6 shows the timing at which processing in each layer is started and the time during which processing is executed in each of the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5. Note that the symbols indicating processing in each layer in FIG. 6 correspond to the layers shown in FIGS. 3 and 4 above. As shown in FIG. 6, the scheduler 132 determines the timing at which processing in each layer is started so that processing in each layer is started after processing in the layer located preceding that layer in each neural network is completed.
[0028] The scheduler 132 also determines the timing so that the first accelerator circuit ACC1 and the third accelerator circuit ACC3 do not simultaneously start access to the semiconductor memory device 111, as in the pooling layer L12 and the convolutional layer L21. In each accelerator device 100, the start of processing in each layer is the peak of access to the semiconductor memory device 111. If multiple processes are simultaneously started in different accelerator devices 100, the amount of access to the semiconductor memory device 111 may exceed the allowable amount, resulting in access to the semiconductor memory device 111 being restricted. Therefore, the scheduler 132 prevents the amount of access to the semiconductor memory device 111 from exceeding the allowable amount by preventing different accelerator circuits from simultaneously starting access to the semiconductor memory device 111. After completing this step, the accelerator device 100 ends the scheduling process.
[0029] The accelerator device 100 of the embodiment described above has the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5, and uses allocation information to determine the timing for starting processing in each layer in the first accelerator circuit ACC1 to the fifth accelerator circuit ACC5. Therefore, processing can be executed in an accelerator circuit having a configuration suitable for processing in each layer, and a decrease in efficiency of arithmetic processing can be suppressed.
[0030] Furthermore, the accelerator device 100 has a layer allocation unit 131 that determines the allocation circuit for each layer using hardware characteristic information and layer characteristic information, so that an accelerator device 100 having a configuration suitable for processing in each layer can be automatically determined.
[0031] Furthermore, the scheduler 132 determines the timing to start processing in each layer so that different accelerator circuits do not simultaneously start accessing the semiconductor memory device 111, thereby preventing the amount of access to the semiconductor memory device 111 from exceeding the allowable amount. This further prevents a decrease in the efficiency of the arithmetic processing.
[0032] B. Other Embodiments: (B1) In the above embodiment, the allocation circuit is determined by the layer allocation unit 131, but the present disclosure is not limited to this. Allocation information indicating the allocation circuit determined by the user may be stored in advance in the register 120. The accelerator device 100 of this type also makes it possible to execute processing in an accelerator circuit having a configuration suitable for processing in each layer, thereby suppressing a decrease in the efficiency of the arithmetic processing.
[0033] (B2) In the above embodiment, the arithmetic circuit 110 includes a buffer unit, but the present disclosure is not limited to this. The first accelerator circuit ACC1 to the fifth accelerator circuit ACC5 may not include a buffer unit if the difference in data processing speed between each cache unit and the arithmetic unit is not large. According to the accelerator device 100 of this embodiment, the configuration of the arithmetic circuit 110 can be simplified, thereby reducing the manufacturing cost of the accelerator device 100. Furthermore, since the register 120 does not need to store the storage capacity of the buffer unit as hardware characteristic information, the amount of information stored by the register 120 can be reduced.
[0034] (B3) In the above embodiment, the scheduler 132 determines the timing so that different accelerator circuits do not simultaneously start accessing the semiconductor memory device 111, but the present disclosure is not limited to this. The scheduler 132 may determine the timing so that different accelerator circuits simultaneously start accessing the semiconductor memory device 111. The accelerator device 100 of this type also makes it possible to execute processing in accelerator circuits having configurations suitable for processing in each layer, thereby suppressing a decrease in efficiency of arithmetic processing.
[0035] (B4) In the above embodiment, the arithmetic unit, buffer unit, and cache unit are configured on the same chip, separate from the storage unit. However, the present disclosure is not limited to this. For example, the functional units functioning as the third accelerator circuit ACC3 to the fifth accelerator circuit ACC5 may be configured on a chip separate from the chip on which the functional units functioning as the first accelerator circuit ACC1 and the second accelerator circuit ACC2 are configured. The accelerator device 100 of this configuration also achieves the same effects as the above embodiment.
[0036] (B5) In the above embodiment, processing in the neural network is assigned to each accelerator circuit on a layer-by-layer basis, but the present disclosure is not limited to this. For example, a case will be described in which a first convolution operation, a second convolution operation, and a third convolution operation are performed in a convolution layer that performs a convolution operation on a three-dimensional tensor represented by width W × height H × channel C. Here, the "first convolution operation" is a convolution operation on C matrices with a matrix size of W × H. The "second convolution operation" is a convolution operation on H matrices with a matrix size of W × C. The "third convolution operation" is a convolution operation on C matrices with a matrix size of W × H. In this case, the accelerator device 100 may further include a plurality of accelerator circuits configured to perform the first to third convolution operations, respectively, and may perform the operations by assigning the first to third convolution operations to each accelerator circuit.
[0037] In addition, an example will be described in which operations using multiple filters with different filter sizes are performed in the convolution layer and the pooling layer. In this case, the accelerator device 100 may further include multiple accelerator circuits configured to perform operations using filters with preset filter sizes, and each operation may be assigned to each accelerator circuit according to the filter size. According to the accelerator device 100 of the above-described embodiment, compared to a configuration in which operation processing is assigned to accelerator circuits on a layer-by-layer basis, each operation can be assigned to an accelerator circuit more suited to the type of operation, thereby further suppressing a decrease in the efficiency of the operation processing.
[0038] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features in each embodiment corresponding to the technical features in the form described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.
[0039] The accelerator device 100 and the methods described herein may be implemented by a special-purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the accelerator device 100 and the methods described herein may be implemented by a special-purpose computer configured with a processor comprising one or more dedicated hardware logic circuits. Alternatively, the accelerator device 100 and the methods described herein may be implemented by one or more special-purpose computers configured with a processor and memory programmed to perform one or more functions in combination with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium. [Explanation of symbols]
[0040] DESCRIPTION OF SYMBOLS 11...first memory unit, 12...first cache unit, 14...first product-sum operation unit, 21...second memory unit, 22...second cache unit, 24...second product-sum operation unit, 31...third memory unit, 32...third cache unit, 34...pooling operation unit, 41...fourth memory unit, 42...fourth cache unit, 44...activation operation unit, 51...fifth memory unit, 52...fifth cache unit, 54...vector operation unit, 100...accelerator device, 111...semiconductor memory device, 120...register, 132...scheduler, ACC1...first accelerator circuit, ACC2...second accelerator circuit, ACC3...third accelerator circuit, ACC4...fourth accelerator circuit, ACC5...fifth accelerator circuit
Claims
1. An accelerator device (100) used in arithmetic processing using a neural network, a plurality of accelerator circuits (ACC1, ACC2, ACC3, ACC4, ACC5) each having an arithmetic unit (14, 24, 34, 44, 54), a cache unit (12, 22, 32, 42, 52), and a memory unit (11, 21, 31, 41, 51) realized by a semiconductor memory device (111); a register (120) for storing assignment information indicating an assigned circuit, which is an accelerator circuit among the plurality of accelerator circuits, each configured to execute a predetermined type of calculation, and which is set in correspondence with each of a plurality of calculation processes executed in the neural network; a scheduler (132) that uses the allocation information to determine timings for starting each of the plurality of arithmetic processes in each of the accelerator circuits; Equipped with Accelerator device.
2. 2. The accelerator device of claim 1, the assigned circuit is an accelerator circuit among the plurality of accelerator circuits that is determined to execute arithmetic processing in each layer constituting the neural network, and is set in correspondence with each of the layers, Accelerator device.
3. 3. The accelerator device according to claim 2, the register further stores hardware characteristic information indicating a processing capacity of each of the accelerator circuits and layer characteristic information indicating a type of arithmetic processing and a processing load in each of the layers; The accelerator device further includes a layer assignment unit (131) that sets the assignment destination circuit for each layer using the hardware characteristic information and the layer characteristic information. Accelerator device.
4. 4. The accelerator device according to claim 3, The register stores, as the hardware feature information, the type and processing capacity of the arithmetic processing by the arithmetic unit in each of the accelerator circuits, the storage capacities of the cache unit and the storage unit, the data transfer rate between the arithmetic unit and the cache unit, and the data transfer rate between the cache unit and the storage unit. Accelerator device.
5. 5. The accelerator device according to claim 1, the scheduler determines the timing so that different accelerator circuits do not simultaneously start accessing the semiconductor memory device. Accelerator device.
Citation Information
Patent Citations
Systems and methods for implementing neural networks in integrated circuits
JP2020534602A
Machine learning accelerator mechanism
US20190205737A1
Maximizing resource utilization of neural network computing system
US20200301739A1
Task scheduling for machine-learning workloads
WO2021101617A1