Information processing device and information processing method

Abstract

The present invention provides an information processing device and an information processing method. The information processing device includes: a control unit configured to set a shift amount based on the bit width of data for each layer of a network including multiple layers; multiple MAC (multiplication-accumulation) units configured to perform multiplication-accumulation operations on multiple data and multiple filter coefficients of the layer; multiple shift operation units configured to shift multiple MAC operation results obtained by the multiple MAC units based on the shift amount; and an addition unit configured to calculate the sum of the multiple MAC operation results shifted by the multiple shift operation units.

Description

Technical Field

[0001] The present invention relates to an information processing device and an information processing method, and more particularly to an arithmetic technique in a network having a plurality of layers. Background Art

[0002] In recent years, the accuracy of image recognition is improving due to the development of deep learning. Convolutional neural network (CNN) is known as a method used in deep learning. In CNN, multiple layers are hierarchically connected, and multiple feature images are included in each layer. Figure 2 An example of a network is shown in which there are four layers (layers 1-4) and four feature images in each layer. Figure 2 In , feature image (i, j) represents the jth feature image in layer i. The filter processing result is calculated by using the learned filter coefficients and feature image pixels (feature data). The filter processing is a multiply-accumulate (MAC) operation and includes multiple multiplications and accumulations. Figure 2 The various arrows shown represent MAC operations.

[0003] The feature image of the current layer is calculated by using the feature image of the previous layer and the filter coefficient corresponding to the previous layer. In order to calculate one feature image of the current layer, information of multiple feature images of the previous layer is required. The equation for the convolution operation used to calculate each feature image of the current layer is as follows.

[0004]

[0005] Among them, i,j (n) is a variable representing the MAC operation result corresponding to the position (i, j) in the nth feature image in the current layer. In equation (1), there are M feature images in the previous layer, and I i,j (m) represents the feature data at position (i, j) in the mth feature image. There are X×Y filter coefficients C 1,1 (m,n) to C X,Y (m,n), and the filter coefficients of each feature image are different. The MAC operation for calculating the nth feature image in the current layer is performed M×X×Y times. After performing the convolution operation, the MAC operation result O is used to i,j (n) Perform processing such as activation and pooling to calculate the feature image of the current layer.

[0006] Since CNN requires a large number of MAC operations, when CNN is applied to embedded systems such as mobile terminals and vehicle-mounted equipment, a highly efficient data parallel processing device is required. Since reducing the bit width of the processed data will reduce the cost of the arithmetic operation unit that calculates the convolution operation result, the degree of parallelism (DOP) of the arithmetic operation unit can be increased. In Y. Li, et al., A 7.663-TOPS 8.2-W Energy-efficient FPGA Accelerator for Binary Convolutional Neural Networks, Proceedings of the 2017 ACM / SIGDA International Symposium on Field-Programmable Gate Arrays, Pages 290-291, Feb. 2017, a hardware layout for processing networks with different data bit widths at each layer is proposed.

[0007] In the method described in Y. Li, et al., A 7.663-TOPS 8.2-W Energy-efficient FPGA Accelerator for Binary Convolutional Neural Networks, Proceedings of the 2017 ACM / SIGDA International Symposium on Field-Programmable Gate Arrays, Pages 290-291, Feb. 2017, different types of arithmetic operators are used to process CNNs with different bit widths in each layer. When the bit width of the feature data of the input layer is 8 bits and the bit width of the feature data of the intermediate layer is 2 bits, a convolution operation unit dedicated to 8-bit data and a convolution operation unit dedicated to 2-bit data are required.

[0008] Although the 8-bit data layer and the 2-bit data layer can be processed in parallel by pipeline processing, the hardware usage efficiency is reduced when the computational amount of the convolution processing is different for each layer. In addition, when feature data with a bit width between 2 bits and 8 bits (for example, 4 bits) is to be processed, the efficiency will be reduced because a convolution operation unit dedicated to 8-bit data will have to be used (this is because there is no convolution operation unit for this bit width).

[0009] In the method described in K.Lee, et al., A 502-GOPS and 0.984-mW Dual-Mode Intelligent ADAS SoC With Real-Time Semiglobal Matching and Intention Prediction for SmartAutomotive Black Box System, IEEE Journal of Solid-State Circuits, Vol.52, No.1, Pages 139-150, Jan.2017, a RNN (recurrent neural network) dedicated hardware is proposed, which has a SIMD (single instruction multiple data) structure that can process feature data sets with multiple bit widths. Although 8-bit data, 16-bit data, and 32-bit data can be processed by using the same hardware, this will increase the processing time when calculating the sum of the data output in parallel, because the SIMD command needs to be executed again after the data is temporarily kept in the memory. Summary of the invention

[0010] The present invention provides a technique for achieving highly efficient processing even when data sets with multiple bit widths exist in a multi-layer network.

[0011] According to a first aspect of the present invention, there is provided an information processing device comprising: a control unit configured to set a shift amount based on a bit width of data for each layer of a network comprising a plurality of layers; a plurality of MAC (multiplication-accumulation) units configured to perform multiplication-accumulation operations on a plurality of data and a plurality of filter coefficients of the layer; a plurality of shift operation units configured to shift a plurality of multiplication-accumulation operation results obtained by the plurality of multiplication-accumulation units based on the shift amount; and an addition unit configured to calculate the sum of the plurality of multiplication-accumulation operation results shifted by the plurality of shift operation units.

[0012] According to a second aspect of the present invention, there is provided an information processing device comprising: a control unit configured to set a shift amount based on the bit width of data for each layer of a network comprising a plurality of layers; a plurality of shift operation units configured to shift a plurality of data of the layer based on the shift amount; a plurality of multiplication-accumulation units configured to perform multiplication-accumulation operations on a plurality of filter coefficients and the plurality of data shifted by the plurality of shift operation units; and an addition unit configured to calculate the sum of a plurality of multiplication-accumulation operation results calculated by the plurality of multiplication-accumulation units.

[0013] According to a third aspect of the present invention, there is provided an information processing method, comprising: for each layer of a network comprising a plurality of layers, setting a shift amount based on the bit width of the data; performing multiplication-accumulation operations on a plurality of data and a plurality of filter coefficients of the layer; shifting a plurality of multiplication-accumulation operation results based on the shift amount; and calculating the sum of the shifted plurality of multiplication-accumulation operation results.

[0014] According to a fourth aspect of the present invention, there is provided an information processing method, comprising: for each layer of a network comprising a plurality of layers, setting a shift amount based on the bit width of the data; shifting a plurality of data of the layer based on the shift amount; performing multiplication and accumulation operations on a plurality of filter coefficients and the shifted plurality of data; and calculating the sum of the calculated plurality of multiplication and accumulation operation results.

[0015] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of data processing;

[0017] Figure 2 is a diagram showing an example of the arrangement of a processing target network;

[0018] Figure 3 is a block diagram showing an example of the hardware arrangement of an information processing apparatus;

[0019] Figure 4 is a block diagram showing an example of the arrangement of the data processing unit 305;

[0020] Figure 5 is a block diagram showing an example of the arrangement of the multiplication accumulator 402 and the shift operator 403;

[0021] FIG. 6A to FIG. 6C is a table showing the relationship between data and processing time;

[0022] Figure 7 is a block diagram showing an example of the arrangement of the data processing unit 305;

[0023] Figure 8 is a block diagram showing an example of the arrangement of the multiplication accumulator 702 and the shift operator 701;

[0024] Fig. 9 is a flow chart of the data processing; and

[0025] FIG. 10A to FIG. 10C are diagrams showing examples of 8-bit feature data, 2-bit feature data, and 4-bit feature data, respectively. DETAILED DESCRIPTION

[0026] Embodiments of the present invention will now be described with reference to the accompanying drawings. Note that each embodiment to be described below is an example of a detailed implementation of the present invention, and is a detailed embodiment of the arrangement described in the appended claims.

[0027] [First embodiment]

[0028] First, refer to Figure 3 The block diagram of the present invention describes an example of the hardware configuration of the information processing device according to the embodiment. Computer devices such as PCs (personal computers), tablet terminal devices, smart phones, etc. can be applied as information processing devices. In addition, the information processing device can be an embedded device to be embedded in such a device.

[0029] The input unit 301 is formed of a user interface such as a keyboard, a mouse, a touch panel, or the like, and can input various instructions to the CPU 306 when operated by a user.

[0030] The data storage unit 302 is a large-capacity information storage device such as a hard disk drive device. The data storage unit 302 stores various information to be used in the information processing apparatus (e.g., OS (Operating System)), various computer programs executed by the CPU 306, data to be used when the CPU 306 performs various processes, and the like. The data stored in the data storage unit 302 includes an image to be processed by the image processing unit 309. Note that information that will be described as "known information" below is also stored in the data storage unit 302. The computer programs and data stored in the data storage unit 302 are loaded into the RAM 308 and the like by the CPU 306, the data processing unit 305, and the image processing unit 309, and become processing objects of the CPU 306, the data processing unit 305, and the image processing unit 309.

[0031] Note that the data storage unit 302 may be a storage medium (e.g., a floppy disk, CD-ROM, CD-R, DVD, memory card, CF card, smart media, SD card, memory stick, xD picture card, USB memory, etc.) In this case, the information processing apparatus needs to include a device for reading information from such a storage medium and writing information to the storage medium.

[0032] The communication unit 303 functions as a communication interface for data communication with an external device. It can be arranged so that the communication unit 303 will obtain information required to perform processing in the information processing apparatus from the external device. The communication unit 303 can send the result of the processing performed by the information processing apparatus to the external device.

[0033] The display unit 304 is formed of a liquid crystal screen or a touch panel screen, and can display images, characters, etc. to display the processing results obtained by the CPU 306, the data processing unit 305, or the image processing unit 309. Note that the display unit 304 may be a projection device such as a projector. The input unit 301 and the display unit 304 may be integrated and formed into a device having both an instruction input acceptance function and a display function, such as a touch screen device.

[0034] The data processing unit 305 uses the image written into the RAM 308 by the image processing unit 309, and performs Figure 1 CNN calculation is performed by the processing of the flowchart of , and the obtained calculation result is output to the data storage unit 302, RAM 308, etc. Note that the image to be processed by the data processing unit 305 is not limited to the image written to the RAM 308 by the image processing unit 309, and may be, for example, an image input by other devices. Figure 4 The data processing unit 305 is described.

[0035] The CPU 306 executes various processes by using computer programs and data stored in the ROM 307 or the RAM 308. This allows the CPU 306 to control the overall operation of the information processing apparatus.

[0036] The ROM 307 stores information that does not need to be rewritten, such as setting data of the information processing apparatus, an activation program, etc. The RAM 308 includes an area for storing computer programs and data loaded from the data storage unit 302 and the ROM 307, information received from an external device by the communication unit 303, etc. The RAM 308 includes a work area used by the CPU 306, the data processing unit 305, and the image processing unit 309 to perform various processes. The RAM 308 can provide various areas as appropriate in this manner.

[0037] The image processing unit 309 reads out the image stored in the data storage unit 302 and writes the image into the RAM 308 after performing pixel value range adjustment on each pixel of the image according to an instruction from the CPU 306 .

[0038] The above-mentioned input unit 301 , data storage unit 302 , communication unit 303 , display unit 304 , data processing unit 305 , CPU 306 , ROM 307 , RAM 308 , and image processing unit 309 are connected to the bus 310 .

[0039] Note that the hardware arrangement of the information processing device is not limited to Figure 3 For example, Figure 3The arrangement may be implemented by a plurality of devices. In addition, the information processing apparatus need not always include devices such as the input unit 301, the data storage unit 302, and the display unit 304, and may be arranged so that these devices are connected to the information processing apparatus via a communication path.

[0040] In addition, some or all of the information described as being stored in the RAM 308 may be stored in the data storage unit 302, and some or all of the information described as being stored in the data storage unit 302 may be stored in the RAM 308. Alternatively, it may be arranged so that a part of the RAM 308 is used as the data storage unit 302, or it may be arranged in a virtual manner so that a storage device of a communication partner device of the communication unit 303 is used as a data storage unit via the communication unit 303.

[0041] In addition, despite the Figure 3 Only one CPU 306 is shown in the figure, but the number of CPUs 306 included in the information processing device is not limited to one, and the information processing device may include multiple CPUs. In addition, the data processing unit 305 and the image processing unit 309 may be implemented as hardware or may be implemented as computer programs. In the latter case, these computer programs will be stored in the data storage unit 302, and the functions of the data processing unit 305 and the image processing unit 309 will be performed by the CPU 306 executing the corresponding computer programs.

[0042] Note that, based on the processing result of the data processing unit 305, the CPU 306 performs image processing and / or image recognition on each frame of the moving image obtained from the communication unit 303 or the data storage unit 302. The result of the image processing or image recognition performed by the CPU 306 is stored in the RAM 308 or the data storage unit 302, or is output to an external device via the communication unit 303. In addition, the result of the image processing or image recognition performed by the CPU 306 can be displayed on the display unit 304 as an image or a character, or can be output as an audio in the case where the information processing apparatus has an audio output function.

[0043] <Network to be processed>

[0044] This embodiment uses CNN as the processing object network. Figure 2 An example of the arrangement of the processing target network is shown. Figure 2 The details of the processing target network are as described above. Note that information of the processing target network (such as the calculation amount of each MAC operation, the size of each feature image, the number of feature images, the number of bits of each feature image, etc.) is stored in the data storage unit 302, etc.

[0045] Figure 2 The number of layers in the processing object network shown is four (layers 1-4), and there are four feature images in each layer. As described above, feature image (i, j) represents the jth feature image of layer i. Moreover, the bit width of each feature image of the layer varies according to the layer. The bit width of each feature image of layer 1 is 8 bits, the bit width of each feature image of layer 2 is 2 bits, the bit width of each feature image of layer 3 is 4 bits, and the bit width of each feature image of layer 4 is 8 bits. Since the first layer (layer 1) and the last layer (layer 4) maintain input / output image information, the bit width (8 bits) of the first and last layers tends to be larger than the bit width (2 bits and 4 bits) of the intermediate layers (layers 2 and 3). Each feature image is formed by a plurality of pixels (feature data).

[0046] The calculation (generation) of the feature images of each layer in layers 1 to 4 performed by the data processing unit 305 will be described below. The MAC operation according to the above equation (1) is performed by using the filter coefficient and the 8-bit feature images (1, 1), (1, 2), (1, 3), and (1, 4) of layer 1. Then, as a result of the MAC operation, the 2-bit feature images (2, 1), (2, 2), (2, 3), and (2, 4) of layer 2 are generated.

[0047] Next, a MAC operation according to the above equation (1) is performed using the 2-bit feature images (2, 1), (2, 2), (2, 3), and (2, 4) of layer 2 and the filter coefficients. Subsequently, as a result of the MAC operation, 4-bit feature images (3, 1), (3, 2), (3, 3), and (3, 4) of layer 3 are generated.

[0048] Next, a MAC operation according to the above equation (1) is performed using the 4-bit feature images (3, 1), (3, 2), (3, 3), and (3, 4) of layer 3 and the filter coefficients. Subsequently, as a result of the MAC operation, 8-bit feature images (4, 1), (4, 2), (4, 3), and (4, 4) of layer 4 are generated.

[0049] <Arrangement Example of Data Processing Unit 305>

[0050] An example of an arrangement of the data processing unit 305 is shown in Figure 4. The data memory 401 holds feature data of each feature image of each layer, and the coefficient memory 404 holds filter coefficients. The multiplication accumulator 402 calculates each MAC operation result by performing a MAC operation using the filter coefficients held in the coefficient memory 404 and the feature data held in the data memory 401. The shift operator 403 shifts each MAC operation result obtained by the multiplication accumulator 402, and the adder 406 obtains the “sum of the shifted MAC operation results” by adding a plurality of shifted MAC operation results. Based on the “sum of the shifted MAC operation results” obtained by the adder 406, the processing unit 407 calculates the activation / pooling processing result, and stores the calculated activation / pooling processing result in the data memory 401. The control unit 405 controls the overall operation of the data processing unit 305.

[0051] Will be based on Figure 1 The flowchart of describes the data processing performed by the data processing unit 305. In step S101, the control unit 405 reads out the feature data and filter coefficients of multiple input feature images from the RAM 308, stores the read feature data in the data memory 401, and stores the read feature coefficients in the coefficient memory 404.

[0052] In step S102, the control unit 405 starts a loop of each layer and sets one of the unprocessed layers as a processing target layer. Since layers 1 to 4 are sequentially set as processing targets in this example, layer 1 will be the first processing target layer.

[0053] In step S103, the control unit 405 sets the shift parameter of the shift operator 403 that defines the shift amount according to the layer information. In step S104, the control unit 405 starts a loop of outputting the feature image and sequentially calculates and outputs the feature data. In step S105, the control unit 405 initializes the MAC operation result stored in the adder 406 to set the MAC operation result to zero. The adder 406 has a sum calculation function.

[0054] In step S106, the control unit 405 starts a loop of input feature images and processes the input feature data in sequence. In step S107, under the control of the control unit 405, the multiplication accumulator 402 and the shift operator 403 perform the above-mentioned MAC operation and shift operation respectively. The details of the processing of step S107 will be described later (steps S115 to S117).

[0055] In step S108, the adder 406 adds multiple MAC operation results to obtain the "sum of shifted MAC operation results". In step S109, the control unit 405 determines the completion of the input feature image cycle. If the processing of all input feature images has been completed, the process proceeds to step S110. Otherwise, the process returns to step S107 and starts the processing of the next unprocessed input feature image. In step S110, the processing unit 407 calculates the activation processing result based on the "sum of shifted MAC operation results" obtained by the adder 406 according to the following formula.

[0056]

[0057] In this case, f(·) is the activation function and x is the input data. Although the ReLU (Rectified Linear Unit) is used to implement the activation function in this example, the present invention is not limited to ReLU, and the activation function can be implemented by other non-linear functions or quantization functions. Note that the bit width of the activation processing result will be adjusted as needed.

[0058] In step S111, the processing unit 407 calculates the activation / pooling processing result by performing pooling processing based on the activation processing result according to the layer information. In step S112, the processing unit 407 stores the activation / pooling processing result calculated in step S111 as the feature image of the next layer in the data memory 401.

[0059] In step S113, the control unit 405 determines the completion of the output feature image cycle. If the processing of all output feature images has been completed, the process proceeds to step S114. Otherwise, the process returns to step S105 to start the processing of the unprocessed output feature image.

[0060] In step S114, the control unit 405 determines the completion of the layer cycle. If the processing of all layers has been completed, the processing ends according to Figure 1 the flowchart. Otherwise, the process returns to step S103 and starts the processing of the unprocessed layer.

[0061] <MAC Operations and Shifting Operations>

[0062] The MAC operations and shifting operations in step S107 (steps S115 to S117) will be described. In step S115, in addition to reading out the feature data from the data memory 401 and transmitting the read feature data to the multiply-accumulator 402, the control unit 405 also reads out the filter coefficients from the coefficient memory 404 and transmits the read filter coefficients to the multiply-accumulator 402. The number of filter coefficients and the transmission count will vary according to the bit width of the feature data.

[0063] In step S116, the multiplier accumulator 402 calculates a MAC operation result based on the feature data and the filter coefficient. In step S117, the shift operator 403 shifts the MAC operation result obtained in step S116 based on the shift amount indicated by the shift parameter set in step S103.

[0064] <Detailed Description of Cases with Different Bit Widths>

[0065] This embodiment can process data with different bit widths. Fig. 10A The operations of the multiplier-accumulator 402, the shift operator 403, and the adder 406 are shown in the case where 8-bit feature data is to be processed. Fig. 10B The operations of the multiplier-accumulator 402, the shift operator 403, and the adder 406 are shown in the case where 2-bit feature data is to be processed.

[0066] In the case where the feature data is 8 bits, the multiplier accumulator 402 divides the 8-bit feature data 1001 (value: 234) into 2-bit data sets (2-bit data), such as Fig. 10A As shown. The multiplier accumulator 402 uses the four 2-bit data sets (values: 2, 2, 2, 3) obtained according to the division and the shared filter coefficient to calculate four MAC operation results, and the shift operator 403 shifts the four MAC operation results based on four shift parameters. Subsequently, the adder 406 adds the four shifted MAC operation results to calculate a feature data set (MAC operation result of 8-bit input feature data). The data processing unit 305 can process an 8-bit input feature data set in this way.

[0067] In the case where the feature data is 2 bits, the multiplier accumulator 402 uses four 2-bit data sets 1002 (values: 2, 2, 2, 3) and four filter coefficients to calculate four MAC operation results, such as Fig. 10B As shown. The shift operator 403 shifts the four MAC operation results based on a shift parameter. Since the shift parameter is zero, the state of the MAC operation result is the same before and after the shift operation. Subsequently, the adder 406 calculates a feature data set (the sum of the MAC operation results of the four 2-bit input feature data sets) by adding the four MAC operation results. The data processing unit 305 can process four 2-bit input feature data sets in parallel in this way.

[0068] Let M be the number of input feature images and 1×1 be the filter size. Since the filter size is equal to one pixel and the values ​​of the variables x and y are constants, we will use I i , j (n) Calculate O i , j(n) The calculation of MAC operation (Equation 1) can be simplified as follows.

[0069]

[0070] Although the multiplier accumulator 402 will calculate the individual convolution results of the filter coefficients and the input feature data when the filter size is greater than 1×1, the multiplier accumulator 402 will calculate the product of I(m) and C(m, n) when the filter size is equal to 1×1.

[0071] Assume that there are two types of feature processing target data, ie, feature processing data having a bit width of α bits and feature processing data having a bit width of β bits. Figure 4 The multiplier accumulator 402 shown includes P α-bit data MAC operation units for calculating MAC operation results, and the shift operator 403 includes P α-bit data shift operation units for calculating shift operation results. α, β and P satisfy the following conditions.

[0072] β=α×P (4)

[0073] When inputting feature data I' (β) When the bit width is β bits, based on the premise of equations (6), (7) and (8), the output of adder 406 is expressed by the following equation (5). The MAC operation result O(n) of the nth output image is given by the following equation.

[0074]

[0075] Among them, I (α),P (m) is the input data of the α-bit data MAC operation unit, C p (m,n) is the filter coefficient, S(p) is the shift parameter. The variable m is the number of the α-bit input feature image group (1 group = P images) (the processing number of the multiplication accumulator 402), the variable p is the MAC operation unit number and the shift operation unit number, and the variable n is the output feature image number. The shift operation is represented by the power of 2 process.

[0076] As shown in equation (6), the filter coefficient C p (m,n) is the filter coefficient C'(m,n) corresponding to the mth β-bit feature image. Since the shared filter coefficients are used for the α-bit input feature image group, the variable p can be omitted. The number of filter coefficients provided in parallel to the P MAC operation units is 1, and the transmission count is 1.

[0077]

[0078] In this case, the input data I' (β) is divided into P α-bit data sets I (α),P(m) Based on the MAC operation unit number p and the bit width α of the divided data, the value of the shift parameter S(p) is calculated by the following formula.

[0079] S(p)=α×(p-1) (7)

[0080] β-bit input feature data I' (β) The P α-bit data sets I are divided into (α),P (m) is expressed as the following formula.

[0081]

[0082] In this case, substituting equations (6), (7) and (8) into equation (5), the equation for output data O(n) is as follows.

[0083]

[0084] On the other hand, when inputting feature data I' (α) When the bit width of is α bits, based on the premise of equations (11), (12) and (13), the output of adder 406 is expressed as (10). The MAC operation result O(n) of the nth output image is given by the following equation.

[0085]

[0086] Among them I (α),P (m) is the input data of the α-bit data MAC operation unit, C p (m,n) is the filter coefficient, S(p) is the shift parameter. The variable m is the number of the α-bit input feature image group (1 group = P images) (the processing number of the multiplication accumulator 402), the variable p is the MAC operation unit number and the shift operation unit number, and the variable n is the output feature image number. The shift operation is represented by the power of 2 process.

[0087] Filter coefficient C p (m, n) is the filter coefficient C′((m-1)×P+p, n) corresponding to the {(m-1)×P+p}th α-bit feature image. Since the filter coefficient is different according to the MAC operation unit number p, the number of filter coefficients to be provided in parallel to the P MAC operation units is P, and the transmission count is P.

[0088]

[0089] Input feature data I′ (α) Becomes the input data I of the α-bit data MAC operation unit (α)，P (m), and the value of the shift parameter S(p) is always 0, as shown in the following formula.

[0090] S(p)=0 (12)

[0091] Although P α bits are input into the feature data set I′ (α) Directly input to the MAC operation unit, but the P input data sets are feature data of different feature images. The feature image number is represented by the MAC operation unit number p, the number of shift operation units P and the processing number m of the multiplication accumulator 402, as shown in the following equation (13).

[0092]

[0093] Substituting equations (11), (12) and (13) into equation (10) yields the following equation for the output data O(n).

[0094]

[0095] By changing the value of the shift parameter S(p) and the number of filter coefficients, the feature data I′ with a bit width of α bits can be processed by using the same operator (multiplier accumulator 402, shift operator 403 and adder 406). (α) and feature data I′ with a bit width of β bits (β) .

[0096] <Processing Example of Case with Different Bit Widths>

[0097] Figure 5 and Fig. 10A and Fig. 10B An example of arrangement is shown when P = 4, β = 8, and α = 2. The bit width of input data to the multiplier accumulator 402 is 2 bits, the bit width of input data to the shift operator 403 is 6 bits, and the bit width of input data to the adder 406 is 12 bits.

[0098] FIG. 6A to FIG. 6C Shown by using Figure 5 The hardware arrangement shown is used to handle Figure 2 An example of processing time in the case of the processing target network shown. Fig. 6A and Fig. 10A An example is shown when layer 1 (8-bit data, number of input feature images M=4) is processed. Feature data I' of feature image (1, 1) (8) (1) is 8 bits, and the four data sets I obtained by dividing the feature data into four parts based on equation (8) (2),1 (1) to I (2),4(1) is input to the multiplication accumulator 402. The shift operation result is calculated by using the input feature data set, the shift parameter and the filter coefficient C (m, n), the calculated shift operation result is input to the adder 406, and the initial value zero is added to the obtained result. The calculation result is set as the shift operation result and is held by the adder 406. The duration of this process is 1 ms.

[0099] Feature data I' of feature image (1, 2) (8) (2) is 8 bits, and the four data sets I obtained by dividing the feature data into four parts based on equation (8) (2),1 (2) to I (2),4 (2) is input to the multiplication accumulator 402. The shift operation result is calculated by using the input feature data set, shift parameter and filter coefficient C(m,n), and the calculated shift operation result is input to the adder 406 and added to the previous result. The duration of this process is 1 ms.

[0100] The feature images (1, 3) and (1, 4) are processed in sequence in a similar manner to the feature image (1, 2), the shift operation results are accumulated, and the addition result is calculated. The duration of this process is 2 ms. Finally, the feature data of the feature image (2, 1) is output via the processing unit 407. The processing time of the four feature images is 4 ms.

[0101] Figure 6B and Fig. 10B An example is shown when layer 2 (2-bit data, number of input feature images M=4) is processed. Each feature data set I' of feature images (2, 1) to (2, 4) (2) (1) to I' (2) (4) is 2 bits, four data sets I (2),1 (1) to I (2),4 (1) is input to the multiplication accumulator 402 in parallel based on equation (13). The shift operation result is calculated by using the input feature data, the shift parameter and the filter coefficient Cp (m, n), the obtained result is input to the adder 406 and added to the initial value zero, and the calculation result becomes the shift operation result. Finally, the feature data of the feature image (3, 1) is output via the processing unit 407. The processing time of the four feature images is 1 ms.

[0102] like Fig. 6A and Figure 6B as well as Fig. 10A and Fig. 10B As shown, when the input feature data is 8 bits, the processing time of each output data is 4 ms, and when the input feature data is 2 bits, the processing time of each output data is 1 ms. The common data processing unit 305 can efficiently process data of different bit widths.

[0103] [Second embodiment]

[0104] The following will describe the differences from the first embodiment. Matters not particularly mentioned below are similar to those of the first embodiment.

[0105] <Order of shift operations and MAC operations>

[0106] The first embodiment describes an example in which a shift operation is performed after a MAC operation. However, even if the order of the MAC operation and the shift operation is switched, the same processing result can be obtained. When the order of the MAC operation and the shift operation is switched, Figure 1 Part of the flowchart will be changed. Step S107 is changed to Fig. 9 Steps S901 to S903 in .

[0107] Figure 7 An example of the arrangement of the data processing unit 305 according to this embodiment is shown. The shift operator 701 shifts the feature data stored in the data memory 401 based on the shift parameter, and the multiply accumulator 702 calculates a MAC operation result based on the shifted feature data and the filter coefficient.

[0108] <Shift Operation and MAC Operation>

[0109] The MAC operation and the shift operation (steps S901 to S903) performed in step S107 will be described. In step S901, the control unit 704 reads out the feature data from the data memory 401 and reads out the filter coefficient from the coefficient memory 703. In step S902, the shift operator 701 shifts the feature data based on the shift parameter set in step S103. In step S903, the multiplication accumulator 702 calculates the MAC operation result based on the shifted feature data and the filter coefficient.

[0110] <Detailed description of different bit widths>

[0111] In this embodiment, the shift operator 701 includes P α-bit data shift operation units for calculating the shift operation result, and the multiplication accumulator 702 includes P α-bit data MAC operation units for calculating the MAC operation result. The output of the multiplication accumulator 702 is expressed by the following equation (15), and is equivalent to the output of the shift operator 403 shown in equation (5).

[0112]

[0113] Figure 8An example of the case where P=4, β=8, and α=2 is shown. The bit width of the input data of the shift operator 701 is 2 bits, the bit width of the input data of the multiplication accumulator 702 is 8 bits, and the bit width of the input data of the adder 406 is 12 bits. Since the circuit scale of the shift operator 701 and the circuit scale of the multiplication accumulator 702 are different due to the different bit widths, the overall circuit scale can be reduced by switching the order of the shift operator 701 (shift operator 403) and the multiplication accumulator 702 (multiplication accumulator 402).

[0114] [Third embodiment]

[0115] The first and second embodiments describe an example in which the bit width of the input feature data is α bits (the bit width of each MAC operation unit) and β bits (the product of the bit width of each MAC operation unit and the number of MAC operation units). However, the present invention is not limited to these, and bit widths other than α and β may be used.

[0116] <When the bit width of input feature data is γ bits>

[0117] In this embodiment, input feature data with a bit width of γ bits can be processed. Fig. 10C An example in which the feature data is 4 bits is shown. In the case where the feature data is 4 bits, the multiplier accumulator 402 divides each of the two 4-bit feature data sets 1003 (values: 10, 14) into 2 bits, as shown in FIG. Fig. 10C As shown. The multiplier accumulator 402 uses the four 2-bit data sets (values: 2, 2, 2, 3) obtained by the division and the two filter coefficients to calculate four MAC operation results. The shift operator 403 shifts the four MAC operation results based on two shift parameters. The adder 406 adds the four shifted MAC operation results to calculate a feature data set (the sum of the MAC operation results of the two 4-bit input feature data sets). In this way, the data processing unit 305 can process two 4-bit input feature data sets in parallel. γ is the bit width of the input feature data, and the value of γ is different from the value of β. The definitions of α, β and P are the same as in the first embodiment, and γ, α and P' satisfy the following conditions.

[0118]

[0119] Where γ is less than β, and P is a multiple of P'. (γ) When the bit width is γ bits, based on the premise of equations (18), (19) and (20), the output data O(n) of adder 406 is expressed by the following equation (17). The MAC operation result O(n) of the nth output feature image is given by the following equation.

[0120]

[0121] Among them I (α) , p (m) is the input data of the α-bit data MAC operation unit, C p (m,n) is a filter coefficient, S(p) is a shift parameter. Variable m is the number of the α-bit input feature image group (1 group = P images) (processing number of the multiplication accumulator 402). The MAC operation unit is divided into P / P' sets, the shift operation unit is divided into P / P' sets, and the variable q is the set number of the MAC operation unit. The variable p is the MAC operation unit number and the shift operation unit number in the set, and the variable n is the output feature image number. The shift operation is represented by a power of 2 process.

[0122] Filter coefficient C p,q (m,n) is the filter coefficient C'((m-1)×P / P'+q,n) corresponding to the {(m-1)×P / P'+q}th γ-bit feature image. The filter coefficient is calculated based on the set number q of the MAC operation unit. Since a part of the filter coefficients is shared, the number of filter coefficients to be provided in parallel to the P MAC operation units is P / P', and the transmission count is P / P'.

[0123]

[0124] In this case, the input feature data I' (γ) Divided into P' α-bit data sets I (α),p (m) The shift parameter S(.) is calculated based on the bit width α of the MAC operation unit and the MAC operation unit number p.

[0125] S(p,q)=α×(p-1) (19)

[0126] γ bits of input feature data I' (γ) i is divided into P' α-bit data sets I (α),p,q (m) indicates.

[0127]

[0128] Substituting equations (18), (19) and (20) into equation (17) yields the following equation for the output data O(n).

[0129]

[0130] By setting the value of the shift parameter S(p,q) and the number of filter coefficients, the feature data I′ having a bit width of γ bits can be processed by using the same operators (multiplier accumulator 402, shift operator 403 and adder 406) as in the first embodiment. (γ) .

[0131] <Processing Example of Case with Different Bit Widths>

[0132] Figure 5 and Fig. 10C An example of arrangement when P=4, β=8, and α=2 is shown. FIG. 6A to FIG. 6C Shows that when using Figure 5 The hardware arrangement shown is used to handle Figure 2 The following is an example of the processing time when the target network is processed.

[0133] Figure 6C and Fig. 10C An example is shown in which P'=2, γ=4 and layer 3 (4-bit data, number of input feature images M=4) is processed. Feature data I' of feature images (3, 1) and (3, 2) (4), (1) and I' (4), Each of (2) is 4 bits, and the four data sets I divided based on equation (20) are (2),1 (1) to I (2),4 (1) Input to the multiplication accumulator 402. The shift operation result is calculated by using the input feature data, the shift parameter and the filter coefficient C (m, n), the calculated shift operation result is input to the adder 406, and the initial value zero is added to the calculated result. The calculation result becomes the shift operation result and is held by the adder 406. The duration of this process is 1 ms.

[0134] Feature data I' of feature images (3, 3) and (3, 4) (4), (3) and I' (4), Each of (4) is 4 bits, and the four data sets I divided based on equation (20) are (2),1 (2) to I (2),4 (2) is input to the multiplication accumulator 402. The shift operation result is calculated by using the input feature data, the shift parameter and the filter coefficient C(m,n), and the calculated shift operation result is input to the adder 406, and the result is added to the previous result. The duration of this operation is 1 ms. Finally, the feature data of the feature image (4, 1) is output via the processing unit 407. The processing time of the four feature images is 2 ms.

[0135] In this way, the advantage of this embodiment is that it is highly flexible because it can process feature data other than data with a bit width of α bits (the bit width of each MAC operation unit) or β bits (the product of the bit width α of each MAC operation unit and the number P of MAC operation units).

[0136] [Fourth embodiment]

[0137] Although the first embodiment describes an example in which the activation process is executed by the processing unit 407, the execution of the activation process is not limited to the processing unit 407, and it may be arranged so that other devices (such as the CPU 306) will execute the activation process. This also applies similarly to other processing operations, and the above embodiment only shows an example of the subject of various processes, and a subject different from the subject described in the above embodiment may be used.

[0138] In addition, in the first embodiment, the activation / pooling process is performed according to the layer information. However, depending on the situation, the activation / pooling process may be omitted.

[0139] In addition, although the first to third embodiments describe the case where the filter size (the height and width of each filter) is 1×1, the filter size is not limited to 1×1 and may be other sizes. The numerical values ​​used in the description of the above embodiments are merely examples for more specific explanations and are not intended to limit the numerical values ​​to be used to those described in the above embodiments.

[0140] When the filter size is small, there is an advantage in that the capacity of the memory (coefficient memory 404 or 703) for storing the filter coefficients can be made small. The minimum value set to the filter width and the filter height is 1.

[0141] Furthermore, the first to third embodiments set the number of input feature images to M, and the number of output feature images to N. However, the numerical values ​​applicable to M and N are not limited to specific numerical values. In this way, the numerical values ​​applicable to the above-mentioned various variables are not limited to specific numerical values.

[0142] In addition, although the filter coefficient is stored in the coefficient memory 404 or 703 and the characteristic data is stored in the data memory 401 in the first to third embodiments, the memory for storing the filter coefficient and the characteristic data is not limited to a specific memory. For example, the filter coefficient and the characteristic data may be stored in a memory included in the multiplication accumulator 402 or 702, or may be stored in the RAM 308.

[0143] In addition, the bit width of each filter coefficient is not limited to a specific bit width. In addition, although CNN is used as a processing object network in the first to third embodiments, the processing object network is not limited to CNN, and may be a network in which a plurality of other types of layers are hierarchically connected, such as RNN, MLP (Multi-layer Perceptron), etc.

[0144] Other embodiments

[0145] The embodiments of the present invention may also be implemented by reading and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above-mentioned embodiments, and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions in the above-mentioned embodiments, and the embodiments of the present invention may be implemented by, for example, reading and executing the computer executable instructions from the storage medium by the computer of the system or device to perform one or more functions in the above-mentioned embodiments, and / or controlling the one or more circuits to perform one or more functions in the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a memory of a distributed computing system, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD) TM ), flash memory devices, memory cards, etc.

[0146] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0147] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: a plurality of multiplication-accumulation units configured to perform multiplication-accumulation operations on a plurality of data and a plurality of filter coefficients of a layer of a network including a plurality of layers; a plurality of shift operation units configured to shift a plurality of multiplication-accumulation operation results obtained by the plurality of multiplication-accumulation units by raising the respective multiplication-accumulation operation results to powers of 2 based on a shift amount; a control unit configured to set, for each layer of the network, a shift amount corresponding to a plurality of shift operation units based on a number in the input data and a bit width of the input data; as well as an adding unit configured to calculate a sum of the plurality of multiplication-accumulation operation results shifted by the plurality of shift operation units, The number of the plurality of multiplication-accumulation units corresponds to the number of the plurality of shift operation units.

2. The information processing device according to claim 1, wherein: The plurality of multiply-accumulate units perform multiply-accumulate operations for each of the plurality of layers on the same circuit.

3. The information processing device according to claim 2, wherein: The plurality of multiplication-accumulation units sequentially perform multiplication-accumulation operations of each of the plurality of layers on the same circuit.

4. The information processing device according to claim 3, wherein: The control unit sets a shift amount for each layer to be processed.

5. The information processing device according to claim 1, wherein: The plurality of layers include two or more layers processing data sets having different bit widths from each other.

6. The information processing device according to claim 1, wherein: The control unit switches the transmission count of the filter coefficient according to the bit width of the data of the layer.

7. The information processing device according to claim 1, wherein: The shift operation unit shifts the plurality of multiplication-accumulation operation results based on a plurality of shift amounts.

8. The information processing device according to claim 1, wherein: The plurality of multiplication-accumulation units and the plurality of shift operation units operate in parallel.

9. The information processing device according to claim 1, wherein: The adding unit stores the sum of the predetermined layer as data of the layer after the predetermined layer in the memory.

10. The information processing device according to claim 1, further comprising: An activation unit is configured to perform an activation process on the sum.

11. The information processing device according to claim 10, further comprising: The pooling unit is configured to perform a pooling process on the result of the activation process.

12. The information processing device according to claim 11, wherein: The pooling unit stores a result obtained by performing pooling processing on a predetermined layer in a memory as data of a layer after the predetermined layer. 13 . The information processing device according to claim 9 , further comprising a unit configured to perform at least one of image processing and image recognition on each frame of the moving image based on the data stored in the memory.

14. The information processing device according to claim 1, wherein: The network has different data bit widths for each layer.

15. The information processing device according to claim 1, wherein: The bit width of the data is 2 bits, 4 bits, or 8 bits.

16. The information processing device according to claim 1, wherein: When the filter size is equal to 1×1, the multiply-accumulate unit calculates the product of the data and the filter coefficients.

17. An information processing device, comprising: a plurality of shift operation units configured to shift the plurality of data by raising each of the plurality of data of the layers of the network including the plurality of layers to a power of 2 based on the shift amount; a control unit configured to set, for each layer of the network, a shift amount corresponding to a plurality of shift operation units based on a number in the input data and a bit width of the input data; a plurality of multiplication-accumulation units configured to perform multiplication-accumulation operations on a plurality of filter coefficients and the plurality of data shifted by the plurality of shift operation units; as well as an adding unit configured to calculate a sum of a plurality of multiplication-accumulation operation results calculated by the plurality of multiplication-accumulation units, The number of the plurality of multiplication-accumulation units corresponds to the number of the plurality of shift operation units.

18. An information processing method, comprising: performing a multiplication-accumulation operation on a plurality of data and a plurality of filter coefficients of a layer of a network including a plurality of layers; shifting the obtained plurality of multiplication-accumulation operation results by raising each multiplication-accumulation operation result to a power of 2 based on the shift amount; For each layer of the network, a shift amount corresponding to the shift is set based on the numbers in the input data and the bit width of the input data; as well as calculating the sum of the shifted multiplication-accumulation operation results, The number of units that perform multiplication and accumulation operations corresponds to the number of units that shift multiple multiplication and accumulation operation results.

19. An information processing method, comprising: shifting the plurality of data by raising each of the plurality of data of the layers of the network including the plurality of layers to a power of 2 based on the shift amount; For each layer of the network, setting a shift amount corresponding to the shift based on the number in the input data and the bit width of the input data; performing a multiplication-accumulation operation on a plurality of filter coefficients and the plurality of shifted data; as well as Calculate the sum of the calculated multiplication and accumulation results. The number of units that perform multiplication and accumulation operations corresponds to the number of units that shift multiple multiplication and accumulation operation results.

20. A non-transitory computer-readable storage medium storing a program for causing a computer to function as each unit of the information processing apparatus defined in any one of claims 1 to 17.

Images