Processing device
By designing a data storage management unit and a computing processing unit composed of multiple SRAMs in a convolutional neural network, the problem of complex size and quantity relationship of feature quantities is solved, and efficient processing and updating feature quantities data in LSI is realized, which is suitable for Image recognition for deep learning.
Patent Information
- Application Number
- CN201880094605.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-06-25
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2038-06-25
AI Technical Summary
When using convolutional neural networks for image recognition, the network needs to be increased in order to increase the detection type or improve the detection accuracy, resulting in an increase in the demand for data storage and coefficient storage, making it difficult to install large-capacity memory in large-scale integrated circuits. The size and quantity relationship of the feature quantity graph is complex, making it difficult to optimize the memory size.
A calculation processing device is designed, and a data storage memory management unit composed of multiple SRAMs is used to transform feature quantity map data into coordinates through a counter, and the SRAM area is divided for storage and reading, combining filtering operation, accumulation addition and nonlinear processing. , realize efficient processing and data update of feature quantities graphs.
It realizes that the calculation processing and data updates are seamlessly performed without overlapping the filter sizes, and can be effectively processed even if the size or number of input feature image is increased. It is suitable for LSI memory for deep learning.
Smart Images

Figure CN112334915B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a processing device, and more particularly, to a circuit structure of a processing device that uses a convolutional neural network to perform deep learning. Background Art
[0002] Conventionally, there are computing devices that perform computing using a neural network in which a plurality of processing layers are hierarchically connected. In particular, deep learning using a convolutional neural network (CNN) has been widely used in computing devices that perform image recognition.
[0003] In image recognition based on deep learning using CNN, the input image data (pixel data) is processed in multiple processing layers of CNN in sequence to obtain the final operation result data of the object contained in the image. The processing layers of CNN are roughly classified into Convolution layers that perform convolution processing including convolution operation processing, nonlinear processing, reduction processing (pooling processing), etc., and FullConnect layers that perform full connection processing that multiplies all inputs (pixel data) by coefficients and accumulates and adds them.
[0004] Image recognition based on deep learning using CNN can achieve a high recognition rate. However, in order to increase the types of objects to be detected or improve the accuracy of object detection, the network needs to be enlarged. In this way, the data storage buffer and coefficient storage buffer must become large-capacity, but large-capacity memory cannot be installed in LSI (Large-Scale Integrated circuit).
[0005] Furthermore, in deep learning in image recognition processing, the relationship between the FM (Feature Map) size and the number of FMs (the number of FM faces) in the (K-1)th layer and the Kth layer is often as shown in the following equation, which makes it difficult to optimize when determining the memory size as a circuit.
[0006] FM size [K] = 1 / 4 × FM size [K-1]
[0007] Number of FMs [K] = 2 × Number of FMs [K-1]
[0008] For example, when considering the memory size of a circuit that can support Yolo_v2, which is one of the variations of CNN, if you want to determine it only by the maximum value of the FM size and the number of FMs, about 1GB is required. In fact, since the number of FMs and the FM size are inversely proportional, about 3MB is sufficient in terms of calculation, but it is necessary to calculate and design memory management in detail for each layer.
[0009] Due to such problems, CNN is generally implemented by software processing using a high-performance PC or GPU (Graphics Processing Unit). However, in order to achieve high-speed processing, it is necessary to use hardware to process heavier parts. An example of such hardware installation is described in Patent Document 1.
[0010] Patent document 1 discloses an operation processing device that realizes efficient operation processing by carrying multiple operation blocks and multiple memories. It has multiple operation processing units, each of which has an operation block and a buffer paired with the operation block, performs convolution operation processing in parallel via a relay unit, and sends and receives cumulative addition data between the operation units, thereby generating input for activation processing at one time even if the input network is large.
[0011] Prior art literature
[0012] Patent Literature
[0013] Patent Document 1: Japanese Patent Application Publication No. 2017-151604 Summary of the invention
[0014] Problems to be solved by the invention
[0015] Patent document 1 has a structure in which a calculation block and a data holding unit are paired. For a network with more than the number of calculation blocks, the redundant data holding unit can repeatedly perform cumulative addition operations to cope with it. However, specific control is required, and in particular, a buffer and optimization of control are required in the data holding unit to correspond to the relationship between the number of FMs and the FM size as described above, but this is not considered.
[0016] In view of the above circumstances, an object of the present invention is to provide an arithmetic processing device that can be applied to deep learning and has a memory size that is suitable for being mounted on an LSI.
[0017] Means for solving problems
[0018] The first mode of the present invention is an operation processing device for deep learning of convolution processing and full connection processing, wherein the operation processing device has: a data storage memory management unit, the data storage memory management unit has: a data storage memory, which is composed of more than 3 SRAMs; a counter, which counts the number of input valid data according to input feature map data and transforms it into coordinates; an SRAM write control unit, which transforms the coordinates transformed by the counter into addresses in the data storage memory, and stores the input feature map data in the data storage memory; and an SRAM read control unit, which receives a request from the outside and takes out the input feature map data stored in the data storage memory according to a prescribed method; a coefficient storage memory management unit, which has a coefficient storage memory for storing coefficients, and refers to the state of the data storage memory management unit to take out the coefficients corresponding to the data taken out from the data storage memory management unit from the coefficient storage memory; a data input unit, which obtains the input feature map data from an external memory for storage; a coefficient input unit, which obtains the coefficients from the external memory for storage; and an operation unit, which obtains the input feature map data from the data storage memory in a structure with N parallel inputs and M parallel outputs, and obtains the input feature map data from the data storage memory. The coefficient storage memory obtains the coefficients and performs filtering processing, cumulative addition processing, nonlinear operation processing and pooling processing, wherein N, M≧1 and are positive numbers; a data output unit, which connects the M parallel data output from the operation unit and outputs them as output feature map data to the storage external memory; and a controller, which controls the data storage memory management unit, the coefficient storage memory management unit, the data input unit, the data output unit and the operation unit, the counter generates the ID of the input feature map data, and the SRAM write control unit controls the data that constitutes the data storage memory. The SRAM is virtually divided into a plurality of areas, and the area to be written is switched according to the ID, and different input feature map data of the same coordinates are stored in the same SRAM, and the SRAM read control unit switches the address to access the area corresponding to the ID of the input feature map data to be retrieved, and the operation unit has: a filter operation unit, which performs filter operation in N parallel manner; a first adder, which accumulates and adds all the operation results of the filter operation unit; a second adder, which accumulates and adds the accumulated addition results of the first adder in the subsequent stage; a trigger, which holds the accumulated addition result of the second adder;and a calculation control unit that generates an ID of input feature map data, obtains the coefficient having the same ID as the input feature map data input from the SRAM read control unit from the coefficient storage memory management unit, and fetches a plurality of input feature map data stored in one data storage memory, and controls the second adder to cumulatively add the cumulative addition results of the first adder until all the input feature map data are fetched. ;
[0019] Alternatively, the operation unit performs operation processing on each input feature quantity map data of the same coordinate, and the SRAM write control unit opens the area storing the input feature quantity map data of the same coordinate and stores subsequent data of the input feature quantity map data whenever the operation processing of the input feature quantity map data of the same coordinate is completed.
[0020] Alternatively, the SRAM read control unit reads the input feature map data from the data storage memory to perform arithmetic processing on m output feature map data to be output based on one input feature map data, and the data output unit counts the valid data to identify which output feature map data it is, appropriately generates an address in the output feature map data, and outputs the output feature map data to the external memory for storage.
[0021] Alternatively, the SRAM read control unit reads partial data of the input feature map data from the data storage memory, and the calculation unit performs calculation processing on the partial data of the input feature map data input from the SRAM read control unit to generate partial data of the output feature map data, and after the processing of the partial data of the input feature map data is completed, enters the processing of the next partial data of the input feature map data.
[0022] Effects of the Invention
[0023] According to the operation processing device of each mode of the present invention, even if the iFM (input feature map) size or the number of iFMs (the number of faces of iFM) increases and cannot completely fit into the size of the data storage memory (IBUF), it is possible to seamlessly implement operation processing and data updating without overlapping the amount of filter size. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 This is a block diagram showing the structure of a calculation processing device according to an embodiment of the present invention.
[0025] Figure 2 This is a diagram showing the configuration of a calculation unit of a calculation processing device according to an embodiment of the present invention.
[0026] Figure 3This is a diagram showing a structure of an IBUF (data storage memory) management unit of the arithmetic processing device according to the embodiment of the present invention.
[0027] Figure 4 This is a diagram showing the structure of an IBUF (data storage memory) of the arithmetic processing device according to the embodiment of the present invention.
[0028] Figure 5 This is a diagram showing how the iFM (input feature map) input to one input ch is divided and stored in IBUF in the arithmetic processing device according to the embodiment of the present invention.
[0029] Figure 6 This is a diagram showing how iFM divided and stored in IBUF is processed in the arithmetic processing device according to the embodiment of the present invention.
[0030] Figure 7 This is a diagram showing an image in which the address offset is changed by iFMID when n=2 in the arithmetic processing device according to the embodiment of the present invention.
[0031] Figure 8 This is a diagram showing a control flow of data writing into IBUF in the arithmetic processing device according to the embodiment of the present invention.
[0032] Fig. 9 This is a diagram showing an example of data arrangement and coefficients of iFM in the arithmetic processing device according to the embodiment of the present invention.
[0033] Fig.10 This is a diagram showing a timing chart of convolution processing in the arithmetic processing device according to the embodiment of the present invention.
[0034] Fig.11 This is a diagram showing an example of a control flow of the SRAM read control unit in the arithmetic processing device according to the embodiment of the present invention.
[0035] Fig.12 This is a diagram showing a schematic diagram of CNN processing in the arithmetic processing device according to an embodiment of the present invention.
[0036] Fig.13 This is a diagram showing an example of a control flow of the SRAM read control unit in the arithmetic processing device according to the embodiment of the present invention.
[0037] Fig.14 This is a diagram showing a timing chart of calculations performed in the calculation processing device according to the embodiment of the present invention.
[0038] Fig.15 This is a diagram showing a schematic diagram of output of a DRAM that writes data in the arithmetic processing device according to the embodiment of the present invention. DETAILED DESCRIPTION
[0039] (Implementation Method 1)
[0040] Embodiments of the present invention will be described using the drawings. Figure 1 This is a block diagram showing the structure of a calculation processing device according to an embodiment of the present invention.
[0041] The operation processing device 1 includes a controller 2, a data input unit 3, a coefficient input unit 4, an IBUF (data storage memory) management unit 5, a WBUF (coefficient storage memory) management unit 6, an operation unit 7, and a data output unit 8. The data input unit 3, the coefficient input unit 4, and the data output unit 8 are connected to a DRAM (storage external memory) 9 via a bus 10. The operation processing device 1 generates an output feature quantity map (oFM) based on the input feature quantity map (iFM).
[0042] The IBUF management unit 5 includes a memory (data storage memory, IBUF) for data storage of the input feature map (iFM), a management / control circuit (data storage memory control circuit) of the data storage memory, a counter 51 to be described later, an SRAM write control unit 52, N IBUFs (data storage memories) (0 to N-1), and an SRAM read control unit 53. Each IBUF is composed of three or more SRAMs.
[0043] The IBUF management unit 5 counts the number of valid data in the input data (iFM data) and converts it into coordinates, further converts the coordinates into IBUF addresses (addresses in IBUF), stores the data in the data storage memory, and fetches the iFM data from the data storage memory according to a prescribed method. The IBUF management unit 5 controls so that when multiple FMs are input, the data with the same coordinates enter the same IBUF.
[0044] The WBUF management unit 6 has a coefficient storage memory (coefficient storage memory, WBUF) and a coefficient storage memory management / control circuit (coefficient storage memory control circuit). The WBUF management unit 6 refers to the state of the IBUF management unit 5 and fetches coefficients corresponding to the data fetched from the IBUF management unit 5 from the coefficient storage memory.
[0045] DRAM 9 stores iFM data, oFM data, and coefficients. Data input section 3 obtains input feature map (iFM) from DRAM 9 in accordance with a prescribed method and delivers it to IBUF (data storage memory) management section 5. Data output section 8 writes output feature map (oFM) data to DRAM 9 in accordance with a prescribed method. Specifically, data output section 8 connects M parallel data output from operation section 7 and outputs them to DRAM 9. Coefficient input section 4 obtains coefficients from DRAM 9 in accordance with a prescribed method and delivers them to WBUF (coefficient storage memory) management section 6.
[0046] The operation unit 7 obtains data from the IBUF (data storage memory) management unit 5 and coefficients from the WBUF (coefficient storage memory) management unit 6, and performs data processing such as filtering, cumulative addition, nonlinear operation, and pooling. The controller 2 controls the entire circuit.
[0047] In CNN, several layers of similar processing are performed in multiple processing layers. In the case of using the operation processing device 1 of the embodiment of the present invention, the operation unit 7 has only the data and coefficients required for the processing of one layer and performs the processing of one layer, and the data and coefficients are placed one by one in the DRAM 9. Then, the processing of the necessary number of layers is repeatedly performed. The operation processing device 1 outputs the subject estimation result as the final output data, and the subject estimation result is obtained by processing the final output data using a processor (which may also be a circuit).
[0048] Figure 2 1 is a diagram showing the structure of the operation unit 7 of the operation processing device according to the embodiment of the present invention. The number of input channels of the operation unit 7 is N (N is a positive number greater than 1), that is, the input data (iFM data) is N-dimensional, and the N-dimensional input data is processed in parallel (input N parallel).
[0049] The number of output channels of the operation unit 7 is M (M is a positive number greater than 1), that is, the output data is M-dimensional, and the M-dimensional input data is output in parallel (output M in parallel). Figure 2 As shown, in one layer, iFM data (d_0 to d_N-1) and coefficients (k_0 to k_N-1) are input according to each channel (ich_0 to ich_N-1), and one data is output. This process is performed in parallel in M layers, and M data och_0 to och_M-1 are output.
[0050] Thus, the computing unit 7 sets the number of input channels to N and the number of output channels to M, and obtains a parallelism of N × M. The number of input channels N and the number of output channels M can be set (changed) according to the size of the CNN, and therefore are appropriately set in consideration of processing performance or circuit scale.
[0051] The operation unit 7 includes an operation control unit 71, which controls each unit in the operation unit. In addition, the operation unit 7 includes a filter operation unit 72, a first adder 73, a second adder 74, an FF (flip-flop) 75, a nonlinear conversion unit 76, and a pooling processing unit 77 for each layer. There are M such layers.
[0052] The operation control unit 71 inputs the specified data to the filter operation unit 72 by issuing a request to the preceding stage of the operation unit 7. The filter operation unit 72 is configured to filter the input data by executing N parallel operations simultaneously by multipliers and adders internally, and output the results of the filtering operations in N parallel operations.
[0053] The first adder 73 adds all the results of the filtering process in the filtering operation unit 72 executed in N parallel and output. The second adder 74 accumulates and adds the operation results of the first adder 73 input in a time-division manner. The second adder 74 accumulates and adds the operation results of the first adder 73 until all the iFM data are acquired. FF 75 is provided to hold the result of the cumulative addition. The nonlinear conversion unit 76 performs nonlinear operation processing based on the Activate function and the like on the results of the cumulative addition in the second adder 74 and the FF 75. The specific implementation is not particularly specified, but the nonlinear operation processing is performed by, for example, a broken line approximation.
[0054] The pooling processing unit 77 performs pooling processing such as selecting and outputting the maximum value (Max Pooling) and calculating the average value (Average Pooling) from a plurality of data input from the nonlinear transformation unit 76. The nonlinear transformation unit 76 and the pooling processing unit 77 can be omitted by the calculation control unit 71.
[0055] The operation control unit 71 controls the above-mentioned processing performed in each unit in the operation unit 7. According to such a structure, in the operation unit 7, the number of input channels N and the number of output channels M can be set (changed) according to the size of the CNN, and therefore, are appropriately set in consideration of processing performance or circuit scale.
[0056] Figure 3 The diagram shows the structure of the IBUF (data storage memory) management unit 5 of the arithmetic processing device according to the embodiment of the present invention. The IBUF management unit 5 includes a counter 51, an SRAM write control unit 52, N IBUFs (data storage memories) (0 to N-1), and an SRAM read control unit 53.
[0057] When the number of input iFM (input feature map) data exceeds the input parallelism N, the iFM data is sometimes input in a time-division manner, and therefore, an ID (hereinafter referred to as iFMID) is required for identifying which iFM data is input. The iFMID is a representative value determined for the data obtained by the data input unit 3 from the DRAM 9 in burst transfer units, and for example, the iFMID of the first 1 to N planes is changed to 0, the iFMID of the N+1 to 2N planes is changed to 1, and so on.
[0058] The counter 51 generates an ID (iFMID) of the input iFM data, counts the number of valid data (input valid data number) in the input data for each iFMID, converts it into coordinates (iFM coordinates), and outputs it.
[0059] The SRAM write control unit 52 converts the iFM coordinates output from the counter 51 into an IBUF address (address in IBUF), and stores the data (iFM data) in the IBUF. In addition, the SRAM write control unit 52 virtually divides each SRAM constituting the IBUF into a plurality of areas, and switches the area to be written by the iFMID. In addition, the SRAM write control unit 52 controls so that different iFM data with the same coordinates are stored in the same SRAM.
[0060] Furthermore, every time the calculation processing of the iFM data of the same coordinates is completed, the SRAM write control unit 52 opens the area storing the iFM data of the same coordinates and stores the subsequent data of the iFM data.
[0061] The SRAM read control unit 53 receives a request (req) from the outside and fetches data from the IBUF according to a predetermined method. In addition, the SRAM read control unit 53 switches the address so as to access the area in the IBUF corresponding to the iFMID to be fetched.
[0062] The operation control unit 71 of the operation unit 7 generates iFMID and obtains the coefficient of the iFMID that is the same as the iFM data input from the SRAM read control unit 53 from the WBUF to fetch a plurality of iFMs stored in one IBUF. In addition, the operation control unit 71 uses the second adder 74 to cumulatively add the operation results of the first adder 73 until all the iFM data are fetched.
[0063] Figure 4 1 is a diagram showing the structure of an IBUF (data storage memory) of the arithmetic processing device of this embodiment. Each IBUF is composed of three or more SRAMs. Figure 4In the example, one IBUF (IBUF_i) is composed of four SRAMs, and three of them (SRAM0 to SRAM2) are used to fetch the data required for the filter operation. The remaining one SRAM (SRAM 3) is used to write the data obtained from the DRAM.
[0064] The data that is no longer needed after the processing in the calculation unit 7 is completed is opened in sequence in units of each SRAM, and the IBUF as a whole works as a circular buffer. In addition, in this example, the IBUF is composed of 4 SRAMs, but as long as the number of SRAMs is 3 or more, it can be any number. By making the number of SRAMs 3 or more, even if, for example, the read position overlaps (strides) in SRAM1 and SRAM2, it is possible to write to SRAM3 in parallel while the data is being taken out in sequence.
[0065] After the use of SRAM1 is finished, it is opened in advance for writing. Since the writing to SRAM3 is completed during the processing, the writing process is transferred to the already opened SRAM1. During this period, the reading side continues to perform filtering processing using the area of SRAM2 to SRAM3. In this way, by opening the unused areas in sequence and updating the data, seamless processing (sliding window processing) can be performed.
[0066] Next, we will explain the case where the number of iFMs (the number of faces of iFMs) exceeds the input parallelism of the circuit and multiple FMs are input to one IBUF. Now assume that the number of iFMs = N×n (N: input parallelism). n faces of data are input to each input ch. Therefore, the IBUF is divided according to the number of input faces n, and sliding window processing is performed in each divided area. Figure 5 As shown, the n-side data input to each input ch is divided and stored in IBUF.
[0067] Figure 5 This is a diagram showing a state where iFM input to one input ch is divided and stored in IBUF. Figure 6 1 is a diagram showing how iFM divided and stored in IBUF is processed. The diagram shows an example where IBUF is composed of four SRAMs (SRAM1, ..., SRAM4). That is, IBUF is physically divided into four SRAMs.
[0068] Each SRAM (SRAM1, ..., SRAM4) is virtually divided into n parts. So that the data of the same area (coordinates) of iFM is stored in each divided area of SRAM. Specifically, the data iFM_0 to iFM_n-1 corresponding to area A of iFM are stored in the divided area of SRAM1. iFM_n to iFM_2n-1 corresponding to area B of iFM are stored in the divided area of SRAM2. iFM_2n to iFM_3n-1 corresponding to area C of iFM are stored in the divided area of SRAM3. iFM_3n to iFM_4n-1 corresponding to area D of iFM are stored in the divided area of SRAM4.
[0069] That is, the data at the same coordinate position of iFM is stored in one SRAM. This is because the data at the same coordinate position of iFM is taken out at the same time and arithmetic processing is performed, and sliding window processing is performed. In addition, by making the number of SRAMs three or more, even when processing data across two areas (coordinates) of iFM (reading data from SRAM1 and SRAM2), writing to the third SRAM can be performed in parallel.
[0070] exist Figure 6 In the example, first, iFM_0 on the first surface, iFM_1 on the second surface, ..., iFM_n-1 on the nth surface are data at the same coordinate position (area A), and they are stored in the divided area of SRAM1. iFM_n on the first surface, iFM_n+1 on the second surface, ..., iFM_2n-1 on the nth surface are data at the same coordinate position (area B), and they are stored in the divided area of SRAM2. iFM_2n on the first surface, iFM_2n+1 on the second surface, ..., iFM_3n-1 on the nth surface are data at the same coordinate position (area C), and they are stored in the divided area of SRAM3. iFM_3n on the first surface, iFM_3n+1 on the second surface, ..., iFM_4n-1 on the nth surface are data at the same coordinate position (area D), and they are stored in the divided area of SRAM4.
[0071] iFM_0, iFM_1, ..., iFM_n-1 stored in SRAM1 are data at the same coordinate position, and are processed at the same timing. They are not needed at the same timing. The SRAM storing the unnecessary data is opened in sequence, and the first surface iFM_4n, the second surface iFM_4n+1, ..., and the nth surface iFM_5n-1 are stored as the subsequent data of iFM. In this way, the sliding window processing is performed.
[0072] In one SRAM area, the size of each area virtually divided is called "divided area size". The divided area size corresponds to the amount of iFM data that can be stored in each area. Used data can be discarded in units of divided area size, so sliding window processing is performed in units of divided area size to open the used area of SRAM so that subsequent data of iFM can be input.
[0073] Specifically, in Figure 6 In the process, the sliding window processing is performed in the partition area size (the partition area size storing the data of the first surface) of iFM_0 of SRAM1, iFM_n of SRAM2, iFM_2n of SRAM3, and iFM_3n of SRAM4. At the same time, the sliding window processing is performed in the partition area size (the partition area size storing the data of the second surface) of iFM_1 of SRAM1, iFM_n+1 of SRAM2, iFM_2n+1 of SRAM3, and iFM_3n+1 of SRAM4. At the same time, the sliding window processing is performed in the partition area size (the partition area size storing the data of the nth surface) of iFM_n-1 of SRAM1, iFM_2n-1 of SRAM2, iFM_3n-1 of SRAM3, and iFM_4n-1 of SRAM4.
[0074] As described above, in the present invention, a plurality of FMs are processed in a time-division manner in each of a plurality of calculation units, thereby being able to cope with a larger network.
[0075] In addition, iFMID increases to 0, 1, 2, ..., n-1 for each plane, so that the address offset can be changed by using iFMID. Figure 6 That way iFM is configured in IBUF. Figure 7 This is a diagram showing an image in which the address offset is changed by iFMID when n = 2. When n = 1, there is no need to refer to iFMID, so iFMID may be fixed to zero.
[0076] Figure 8 1 is a diagram showing a control flow of writing data to IBUF in the arithmetic processing device of the present embodiment. The iFM segmentation loop starts from step S1. In step S2, it is determined whether IBUF can be written. If IBUF cannot be written, the process proceeds to step S3, stands by (wait), and returns to step S2.
[0077] If IBUF can be written, go to step S4. Start the segmentation area size loop from step S4, and start the iFM number (n) loop from step S5. Perform address calculation in step S6, and determine whether data is input in step S7. If no data is input, go to step S8, standby (wait), and return to step S7.
[0078] If there is data input, proceed to step S9 and write data to IBUF. In step S10, determine whether processing equivalent to the number of iFMs (n) has been performed. If not, return to step S5. If it has been performed, proceed to step S11. In step S11, determine whether processing equivalent to the size of the segmented area has been performed. If not, return to step S4. If it has been performed, proceed to step S12. In step S12, determine whether iFM segmentation has been completed. If not, return to step S1. If it has been completed, end the processing (data writing processing to IBUF).
[0079] Next, the processing when the data input to IBUF is read as described above is described. Multiple iFMs are added to each input ch, but it is assumed that two iFMs (iFM1, iFM2) are input. Furthermore, it is assumed that the data configuration of iFM and the coefficients of the filter applied to it are as follows: Fig. 9 Like that. Fig. 9 This is a diagram showing an example of the data arrangement and coefficients of iFM.
[0080] In iFM1, data configuration is as follows Fig. 9 a1, b1, ..., p1 represent data, and 4×4 data are arranged. The coefficients of the filter applied to each of the 3×3 data (a1, b1, c1, d1, e1, f1, g1, i1, j1, k1) are as follows: Fig. 9 w1, w2, ..., w9 represent the coefficients of each data.
[0081] Similarly, in iFM2, data configuration is as follows Fig. 9 a2, b2, ..., p2 represent data, and 4×4 data are configured. The coefficients of the filter applied to each of the 3×3 data (a2, b2, c2, d2, e2, f2, g2, i2, j2, k2) are as follows: Fig. 9 x1, x2, ..., x9 represent the coefficients of each data.
[0082] Timing diagram Fig.10 shown. Fig.10This is a diagram showing a timing diagram when the number of iFMs = 2N during convolution processing in the operation processing device of this embodiment. The filtering operation result of the 3×3 9 data a1 to k1 and the coefficients w1 to w9 is A, and the filtering operation result of the 3×3 9 data a2 to k2 and the coefficients x1 to x9 is B. The filtering operation results are cumulatively added in the time direction in the second adder, and the results are stored in the FF. The final cumulative addition result in the second adder is Σ.
[0083] That is, iFM1 and iFM2 are input in a time-division manner, and are cumulatively added in the second adder, and the result is held in the FF. When the result of the second adder is A or A+B, it is still in the middle of the cumulative addition, so it is not output to the nonlinear transformation unit. When the cumulative addition in the second adder is completed, the cumulative addition result Σ is output to the nonlinear transformation unit, and the FF returns to the initial value. The nonlinear transformation unit performs a nonlinear transformation f(Σ).
[0084] Fig.11 1 is a diagram showing an example of a control flow of the SRAM read control unit in the arithmetic processing device of the present embodiment. First, in step S21, the oFM_V loop starts. The oFM_V loop is a loop for processing the data of the oFM in the V direction (horizontal). Then, in step S22, the oFM_H loop starts. The oFM_H loop is a loop for processing the data of the oFM in the H direction (vertical). In step S23, a WBUF access address is generated, and in parallel, an IBUF access address is generated in step S26.
[0085] When the WBUF access address is generated in step S23, a data storage determination is performed in step S24. If the data storage determination is OK, the process proceeds to the pooling loop in step S29. If the data storage determination is NG, the process waits in step S25 and returns to step S24.
[0086] In addition, when the IBUF access address is generated in step S26, a data storage determination is performed in step S27. If the data storage determination is OK, the pooling loop of step S29 is entered. If the data storage determination is NG, the process waits in step S28 and returns to step S27.
[0087] After the pooling loop in step S29, the iFM number loop in step S30 and the filter size loop in step S31 are entered, and the processing of the operation unit is performed in step S32. In step S33, it is determined whether each loop can be ended. If it has not been ended, the loop is returned to each loop. If all loops have been ended, the control flow ends.
[0088] The data storage judgment of IBUF is a judgment about whether the coordinate data of all iFMs required for filtering processing including pooling processing has been stored in IBUF, and the innermost loop becomes the after judgment. In addition, it is assumed that the judgment method can be changed in various ways. In addition, as long as the judgment is made in units of the H size of oFM, the oFM_H loop is also placed after the data storage judgment.
[0089] The memory control and memory storage method described in this embodiment can also be applied to an algorithm that generates output data by inputting a plurality of frames, for example, a compression process or a noise reduction process such as MPEG that utilizes inter-frame correlation.
[0090] (Implementation Method 2)
[0091] In Embodiment 1, a circuit for processing a layer of a CNN and how to expand the iFM input to the circuit into an IBUF and process it are described. In Embodiment 2, in particular, it corresponds to the case where the number of oFMs (the number of faces of oFM) exceeds the parallelism M of the circuit.
[0092] Fig.12 This is a diagram showing a schematic diagram of the processing of the CNN according to this embodiment. Fig.12 An example of the number of oFMs = 2M is shown, and only one input ch is shown. Assuming that the number of oFMs exceeds the output parallelism M, the number of oFMs = M×m (m>1). CNN is a process of calculating one pixel of oFM based on the same coordinate data of all iFMs.
[0093] At this time, when oFM is generated for each plane, iFM is processed as a sliding window in IBUF, so the data of the first part is overwritten and no longer exists in IBUF. Therefore, every time a new oFM is generated, iFM must be read again from DRAM.
[0094] Therefore, in this embodiment, control is performed to generate all oFMs that can be processed using the data stored in the IBUF. Specifically, after performing the processing as many times as possible on one oFM from the iFM input to the IBUF (only a part is stored), the processing corresponding to the next oFM is performed without updating the IBUF (coefficients are switched appropriately).
[0095] At this time, "the maximum number of processing times" is efficient when it is set to the segmentation area size as a unit on the IBUF. This is because the data of the iFM1 surface that enters one SRAM constituting the IBUF is the segmentation area size. When processing is performed in units of this segmentation area size, at the moment when the processing of the number of oFMs to be output is completed, the data of one SRAM is used up and can be released. In addition, the segmentation area size is the amount of data in the iFM, which is not necessarily consistent with the amount of data on the oFM. This is because reduction based on pooling processing occurs.
[0096] The reason why oFM is not changed in pixel units is that it is more efficient to transfer a certain amount of data to DRAM. If there is no problem with the buffer size or transfer efficiency, it is also possible to switch in pixel units.
[0097] Thus, in this embodiment, all processing that can be performed with the input data is performed before sliding window processing. In addition, the output data is not completed and outputted as oFMs one by one, but parts of a plurality of oFMs are outputted little by little, and a plurality of oFMs are finally completed and outputted.
[0098] Fig.13 : is a diagram showing an example of the control flow of the SRAM read control unit in the arithmetic processing device of the present embodiment. The "dM counter" is equivalent to a loop counter of the segmented area size. First, in step S41, it is assumed that the dM counter = 0. Then, the oFM_V loop starts from step S42, and the oFM_H loop starts from step S43. The oFM_V loop is a loop for processing the data of oFM in the V direction (horizontal), and the oFM_H loop is a loop for processing the data of oFM in the H direction (vertical).
[0099] In step S44, a WBUF access address is generated, and in parallel, an IBUF access address is generated in step S47.
[0100] When the WBUF access address is generated in step S44, a data storage determination is performed in step S45. If the data storage determination is OK, the process proceeds to a pooling loop in step S50. If the data storage determination is NG, the process waits in step S46 and returns to step S45.
[0101] Furthermore, when the IBUF access address is generated in step S47, a data storage determination is performed in step S48. If the data storage determination is OK, the process proceeds to the pooling loop in step S50. If the data storage determination is NG, the process waits in step S49 and returns to step S48.
[0102] After the pooling loop in step S50, the iFM number loop in step S51 and the filter size loop in step S52 are entered, and the processing of the operation unit is performed in step S53. In steps S54 and S55, it is determined whether each loop can be ended. If it has not been ended, the loop is returned to each loop. If all loops have been ended, it enters step S56.
[0103] In step S56, it is determined whether the dM counter is equal to or larger than the segmented region size. If the dM counter is equal to or larger than the segmented region size, the process proceeds to step S57, where the dM counter is set to 0. Then, in step S58, the coefficients are sent to the next oFM, and the process proceeds to step S60.
[0104] When the dM counter is not equal to or greater than the divided region size (when the dM counter is smaller than the divided region size), the process proceeds to step S59 to increase the value of the dM counter, and then proceeds to step S60.
[0105] In step S60 , it is determined whether the oFM_V loop and the oFM_H loop can be terminated. If they have not been terminated, the control flow returns to each loop. If all the loops have been terminated, the control flow ends.
[0106] Fig.14 An example of a timing chart is shown in the case where the number of iFMs = 2N and the number of oFMs = 3M in the arithmetic processing device of this embodiment. When the partition region size is 4D, since pooling exists, the number of data on the oFM is the number of oFMs × D.
[0107] The data output unit counts the number of valid data after pooling, and can identify which oFM is currently received, so that the data is appropriately developed on the DRAM based on the identification result. Fig.15 1 is a diagram showing a schematic diagram of the output of a DRAM when writing data. The numbers in the diagram indicate the output order.
[0108] Thus, in this embodiment, the SRAM read control unit in the IBUF management unit takes out data from the IBUF to perform filtering processing on the m oFMs to be output based on the same input data. Then, the data output unit counts the valid data to identify which oFM data it is, generates an address appropriately, and outputs it to the DRAM. Thus, it is possible to output the oFM number exceeding the output parallelism without re-reading the iFM.
[0109] That is, the SRAM read control unit reads out partial data of the iFM data from the IBUF. The operation unit performs operation processing on the partial data of the iFM data input from the SRAM read control unit (processes the data at the read position) to generate partial data of the oFM data. Then, after the processing is completed, the next partial data of the iFM data is processed.
[0110] An embodiment of the present invention has been described above, but the technical scope of the present invention is not limited to the above embodiment, and the combination of constituent elements may be changed, or various changes or deletions may be made to each constituent element within the scope of the present invention.
[0111] In this specification, words indicating directions such as "front, back, up, down, right, left, vertical, horizontal, row, column" are used to describe these directions in the device of the present invention. Therefore, these words used for description in the specification of the present invention should be interpreted relatively in the device of the present invention.
[0112] Industrial Applicability
[0113] The present invention can be widely used in operation processing devices that perform deep learning using convolutional neural networks. Even if the iFM size or the number of iFMs is too large to fit completely into the size of the data storage memory (IBUF), operation processing and data updating can be seamlessly performed without overlapping the amount of filter size.
[0114] Description of symbols
[0115] 1: operation processing device; 2: controller; 3: data input unit; 4: coefficient input unit; 5: IBUF (data storage memory) management unit; 6: WBUF (coefficient storage memory) management unit; 7: operation unit; 8: data output unit; 9: DRAM (external memory for storage); 10: bus; 51: counter; 52: SRAM write control unit; 53: SRAM read control unit; 71: operation control unit; 72: filter operation unit; 73: first adder; 74: second adder; 75: FF (flip-flop); 76: nonlinear transformation unit; 77: pooling processing unit.
Claims
1. A computing processing device for deep learning of convolution processing and full connection processing, in, The computing device has: A data storage memory management unit, the data storage memory management unit comprising: A data storage memory, which is composed of more than 3 SRAMs; a counter that counts the number of input valid data according to the input feature quantity map data and converts it into coordinates; An SRAM writing control unit, which transforms the coordinates transformed by the counter into addresses in the data storage memory, and stores the input feature quantity map data in the data storage memory; as well as An SRAM readout control unit, which receives a request from the outside and fetches the stored input feature map data from the data storage memory according to a prescribed method; a coefficient storage memory management unit having a coefficient storage memory for storing coefficients, and taking out from the coefficient storage memory the coefficients corresponding to the data taken out from the data storage memory management unit with reference to the state of the data storage memory management unit; a data input unit that acquires the input feature quantity map data from an external memory for storage; A coefficient input unit that obtains the coefficient from the external memory for storage; An operation unit, which obtains the input feature map data from the data storage memory and the coefficients from the coefficient storage memory in a structure of N parallel inputs and M parallel outputs, and performs filtering processing, cumulative addition processing, nonlinear operation processing and pooling processing, wherein N and M are ≥ 1 and are positive numbers; A data output unit that connects the M parallel data output from the calculation unit and outputs the data to the storage external memory as output feature quantity map data; and a controller that controls the data storage memory management unit, the coefficient storage memory management unit, the data input unit, the data output unit, and the operation unit, The counter generates an ID of the input feature quantity map data inputted, The SRAM write control unit controls to virtually divide each of the SRAMs constituting the data storage memory into a plurality of areas, switches the area to be written according to the ID, and stores different input feature map data of the same coordinates in the same SRAM, The SRAM read control unit switches the address so as to access the area corresponding to the ID of the input feature quantity map data to be retrieved, The computing unit comprises: A filter operation unit, which performs filter operations in N parallel manner; a first adder for accumulating and adding all the operation results of the filter operation unit; a second adder for cumulatively adding the cumulative addition results of the first adder at a later stage; a flip-flop that holds the cumulative addition result of the second adder; and The operation control unit is controlled to generate an ID of the input feature quantity map data, and obtain the coefficient whose ID is the same as the input feature quantity map data input from the SRAM read control unit from the coefficient storage memory management unit to retrieve multiple input feature quantity map data stored in a data storage memory, and is controlled so that the second adder accumulates and adds the cumulative addition results of the first adder until all the input feature quantity map data are retrieved.
2. The operation processing device according to claim 1, in, The calculation unit performs calculation processing for each input feature quantity map data of the same coordinate. The SRAM write control unit opens the area storing the input feature amount map data of the same coordinates and stores subsequent data of the input feature amount map data every time the calculation process of the input feature amount map data of the same coordinates is completed.
3. The operation processing device according to claim 1, in, The SRAM read control unit reads the input feature map data from the data storage memory to perform arithmetic processing of m output feature map data to be output based on one input feature map data, The data output unit counts the valid data to identify which output feature amount map data it is, appropriately generates an address in the output feature amount map data, and outputs the output feature amount map data to the storage external memory.
4. The operation processing device according to claim 2, in, The SRAM read control unit reads the input feature map data from the data storage memory to perform arithmetic processing of m output feature map data to be output based on one input feature map data, The data output unit counts the valid data to identify which output feature amount map data it is, appropriately generates an address in the output feature amount map data, and outputs the output feature amount map data to the storage external memory.
5. The operation processing device according to any one of claims 1 to 4, in, The SRAM read control unit reads part of the input feature map data from the data storage memory, The calculation unit performs calculation processing on partial data of the input feature amount map data input from the SRAM read control unit to generate partial data of the output feature amount map data. After the processing of the partial data of the input feature amount map data is completed, the processing of the next partial data of the input feature amount map data is started.
