Neural network processing apparatus and neural network processing method

Abstract

To shorten processing time by reducing the volume of data transfer between a computing unit and a memory connected to outside of the computing unit in a neural network processing apparatus that performs convolution processing with integers.SOLUTION: A neural network processing apparatus performs convolution processing to perform a product-sum operation between an input integer and weight data in each of a plurality of layers in a neural network, converts the integer as the output of the convolution processing to a floating point number, performs activation function processing on the basis of the floating point number, performs integer-conversion processing to convert the floating point number as the output of the activation function processing to an integer, and writes the integer into a memory.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates generally to neural network processing. [Background technology]

[0002] As neural network processing devices, for example, devices described in Patent Documents 1 and 2 are known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-116266 [Patent Document 2] Japanese Patent Publication No. 2023-159945 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the inventor's study, the device shown in FIG. 1 can be considered as a first comparative example of a neural network processing device. In FIG. 1, "I" stands for integerization, "C" stands for convolution, and "F" stands for FP (floating-point) conversion. This neural network processing device acquires floating-point input data, quantizes (integerizes) the floating-point input data to obtain quantized input data, and provides the quantized input data to a computation unit. This neural network processing device also includes a quantization unit that performs inverse quantization (FP conversion) on the computation result output by the computation unit to obtain the inverse quantized result. The computation unit performs matrix-vector operations and / or convolution operations on the quantized (integerized) input data to obtain the computation result of the input data. According to this first comparative example, by performing the convolution operation using integers, processing can be performed faster than when the convolution is performed using floating-point numbers.

[0005] According to the inventor's investigation, the neural network processing device shown in FIG. 2 can be considered as a second comparative example. In FIG. 2, "R" denotes reading from memory, "C" denotes convolution, "A" denotes activation function processing, and "W" denotes writing to memory. This neural network processing device performs neural network processing including convolution processing, which generates multiple intermediate data by performing product-sum operations between multiple input data and multiple weight data, and activation function processing, which generates multiple output data by calculating the value of a predetermined function using the multiple intermediate data as input. This neural network processing device performs activation function processing within the convolution processing loop. This second comparative example eliminates the need to store the convolution processing results in memory, thereby reducing the processing time required for memory read / write and enabling faster processing compared to when the convolution processing and the activation function processing are performed using different functions.

[0006] According to the study by the inventors of the present application, it is possible to configure a neural network processing device with a shorter processing time by combining the first comparative example and the second comparative example. In this case, it is necessary to take into consideration the processing contents of the activation function. Figure 3 shows the processing contents of the activation function called mish. mish is an exponential function (e x), when convolution is performed using integers, the activation function cannot be calculated using the convolution result (integer), and the convolution result must be converted to a floating-point number before the activation function is calculated. Therefore, to combine the first and second comparative examples, the processing shown in Figure 4 is required. In the processing of layer n-1, layer n, and layer n+1, the output of the previous layer is read from memory, and integer conversion, convolution, FP conversion (=floating-point conversion), and activation function processing are performed in that order, and the results are written to memory. Here, the data stored in memory (layer n-1 output (FP), layer n output (FP)) are floating-point numbers. In neural network processing, if the processing time of the convolution is reduced by performing the convolution operation using integers compared to when the convolution is performed using floating-point numbers, the proportion of the processing time spent reading and writing memory often becomes relatively large. In the case of the processing contents of Figure 4, the data stored in memory is floating-point numbers, so the amount of data read and written to memory is the same as in the second comparative example, and in this respect, the effect of speeding up by combining it with the first comparative example is limited.

[0007] In order to address this problem, the present invention aims to reduce the amount of data stored in memory, thereby reducing the processing time of a neural network processing device. [Means for solving the problem]

[0008] To achieve the above object, the neural network processing device performs convolution processing in each of a plurality of layers in the neural network, which performs a product-sum operation between input integers and weight data, performs floating-point conversion processing to convert the integers output from the convolution processing into floating-point numbers, performs activation function processing to calculate a predetermined function from the output of the floating-point conversion processing, performs integer conversion processing to convert the floating-point numbers calculated as a result of the activation function processing into integers, and writes the integers obtained by the integer conversion processing to memory as outputs. [Effects of the Invention]

[0009] According to the present invention, the data stored in the memory is integers, reducing the amount of data, thereby reducing the processing time of the neural network processing device. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 10 is a diagram illustrating processing contents of a first comparative example. [Figure 2] FIG. 10 is a diagram illustrating processing contents of a second comparative example. [Figure 3] FIG. 10 is a diagram illustrating the processing content of the mish activation function. [Figure 4] FIG. 10 is a diagram showing processing contents when the first comparative example and the second comparative example are combined. [Figure 5] FIG. 2 is a hardware configuration diagram of a neural network processing device. [Figure 6] FIG. 6 is a diagram illustrating a first example of the configuration of neural network processing executed by the neural network processing device of FIG. 5. [Figure 7] FIG. 7 is a diagram illustrating an example of processing of one layer of the neural network processing of FIG. 6. [Figure 8] FIG. 8 is a diagram illustrating the contents of the integer conversion process in FIG. 7. [Figure 9] 8 is a diagram showing the process flow when the software in FIG. 7 is executed, and data exchange with a memory. [Figure 10] FIG. 7 is a diagram illustrating a second example of the configuration of the neural network processing executed by the neural network processing device of FIG. [Figure 11] FIG. 11 is a diagram showing an example of software for executing the addition process 1003 of the neural network process in FIG. 10. [Figure 12] FIG. 7 is a diagram illustrating a third example of the configuration of the neural network processing executed by the neural network processing device of FIG. [Figure 13] FIG. 13 illustrates an example of software for processing one layer of the neural network processing of FIG. 12. [Figure 14]FIG. 10 is a diagram illustrating an example of variations of software for performing product-sum calculation processing and corresponding parameters. [Figure 15] FIG. 10 is a diagram illustrating variations of software for performing activation function processing. [Figure 16] FIG. 7 is a diagram illustrating an example of a combination of parameters of product-sum operation processing and activation function processing for each layer of the neural network processing in FIG. 6. [Figure 17] FIG. 17 is a diagram showing the results of selecting combinations of parameters and activation functions in FIG. 16, excluding overlaps. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following description, an "interface apparatus" may refer to one or more interface devices, which may be at least one of the following: An I / O interface apparatus is one or more I / O (Input / Output) interface devices. The I / O (Input / Output) interface devices are interface devices for at least one of an I / O device and a remote display computer. The I / O interface device for the display computer may be a communications interface device. The at least one I / O device may be a user interface device, for example, either an input device such as a keyboard and a pointing device, or an output device such as a display device. A communication interface apparatus that is one or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., an NIC and an HBA (Host Bus Adapter)).

[0012] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0013] In the following description, a "persistent storage device" may refer to one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and more specifically, may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVME) drive, or a storage class memory (SCM).

[0014] In the following description, the term "storage device" may refer to at least one memory, including memory and persistent storage device.

[0015] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing (for example, an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)).

[0016] In the following description, when elements of the same type are described without distinction, common reference symbols are used, and when elements of the same type are described with distinction, reference symbols are used. [Example]

[0017] FIG. 5 is a hardware configuration diagram of a neural network processing device.

[0018] To the neural network processing device 508, input / output devices or sensors such as one or more cameras 513 (e.g., 513_1 and 513_2) and an input / output device such as a display 515 are connected as external devices. The neural network processing device 508 has a microprocessor 501, a GPU 511, and memories (e.g., a ROM (Read Only Memory) 509, RAMs (Random Access Memories) 510_1 and 510_2). The microprocessor 501 has a CPU 502 and a plurality of interface circuits (e.g., 503_1 to 503_5). The interface circuit 503_1 is an interface circuit for the ROM 509. The interface circuit 503_2 is an interface circuit for the RAM 510_1. The interface circuit 503_3 is an interface circuit for the GPU 511. The interface circuit 503_4 is an interface circuit for the cameras 513_1 and 513_2. The interface circuit 503_5 is an interface circuit for the display 515.

[0019] The neural network processing device 508 receives image data from the cameras 513_1 and 513_2, performs neural network recognition, and displays the recognition results on the display 515. While the present embodiment illustrates an example of image recognition, the present invention is not limited thereto and can be applied to neural network processing devices in general, such as speech recognition and text analysis. For example, the neural network processing device 508 may be mounted on a train consisting of n cars (n is a natural number) (or may be mounted so as to be able to communicate with the train), and the image recognition results by the neural network processing device 508 may be input to a train control device, which may perform train operation control (e.g., automatic driving control) based on the image recognition results. Furthermore, for example, a train may be an example of a vehicle, and the neural network processing device 508 may be mounted on a vehicle other than a train (e.g., a passenger car or a truck) (or may be mounted so as to be able to communicate with the vehicle), and the image recognition results by the neural network processing device 508 may be input to a vehicle control device, which may perform vehicle operation control (e.g., automatic driving control) based on the image recognition results.

[0020] The cameras 513_1 and 513_2 capture images in different directions. This makes it possible to capture all images of the area to be recognized and perform recognition processing. Furthermore, cameras with a large number of pixels and a wide shooting range may be used as the cameras 513_1 and 513_2. This makes it possible to cover the entire area with a small number of cameras even if the area to be recognized is large. In this case, an image captured by one camera may be divided into multiple areas and recognition processing may be performed separately. This makes it possible to exclude areas that do not need to be recognized, such as the sky, and perform recognition processing only on the necessary areas.

[0021] The microprocessor 501 is an LSI in which a CPU 502 and interface circuits 503_1 to 503_5 are integrated on a single chip. This configuration is an example, and some or all of the ROM 509, RAMs 510_1 and 510_2, and GPU 511 may be built into the microprocessor 501. The RAM 510_1 may be an example of a memory in which calculation results of the microprocessor 501 are stored, and the RAM 510_2 may be an example of a memory in which calculation results of the GPU 511 are stored.

[0022] The CPU 502 retrieves and executes software stored in the ROM 509 via the interface circuit 503_1. Since the ROM 509 is often slower than the RAM 510_1, the software may be copied from the ROM 509 to the RAM 510_1 at startup and then retrieved from the RAM 510_1. The CPU 502 performs the following series of processes in accordance with the software retrieved from the ROM 509 or RAM 510_1.

[0023] The CPU 502 first captures image data from the cameras 513_1 and 513_2 via the interface circuit 503_4, and stores the captured image data in the RAM 510_1 via the interface circuit 503_2.

[0024] Next, the CPU 502 reads out the image data stored in the RAM 510_1 via the interface circuit 503_2, and transfers it to the GPU 511 via the interface circuit 503_3.

[0025] Next, the CPU 502 reads out the software stored in the ROM 509 or RAM 510_1 via the interface circuit 503_1 or 503_2, transfers the read out software to the GPU 511 via the interface circuit 503_3, and instructs the GPU 511 to start calculation.

[0026] Next, when the CPU 502 receives a notice of the completion of the calculation from the GPU 511 via the interface circuit 503_3, it takes in the calculation result from the GPU 511 via the interface circuit 503_3 and stores the calculation result in the RAM 510_1 via the interface circuit 503_2.

[0027] The CPU 502 then takes in the calculation results of the GPU 511 from the RAM 510_1 via the interface circuit 503_2, performs predetermined processing based on the calculation results, and displays the results of the predetermined processing on the display 515 via the interface circuit 503_5.

[0028] When the GPU 511 receives image data from the CPU 502 via the interface circuit 503_3, the GPU 511 stores the image data in the RAM 510_2.

[0029] Furthermore, when the GPU 511 receives software from the CPU 502 via the interface circuit 503_3, it executes the software and stores the calculation results in the RAM 510_2.

[0030] Furthermore, when the GPU 511 receives a read request for the calculation result from the CPU 502 via the interface circuit 503_3, it reads the calculation result from the RAM 510_2 and outputs the calculation result to the interface circuit 503_3.

[0031] Fig. 6 is a diagram showing a first example of the configuration of neural network processing executed by the neural network processing device 508 in Fig. 5. Note that this diagram describes recognition processing for a single input image, and when performing recognition processing for multiple images, the processing in Fig. 6 is executed for each input image. In this case, the processing content may be the same for all images, or may be different for each image according to the purpose of the recognition processing.

[0032] The processing in Figure 6 is composed of N layers 901. Layer 1 (901_1) receives an input image and outputs the processing result to layer 2 (901_2). Layer N (901_N) receives data from the previous layer and outputs the processing result as the recognition result. The other layers receive data from the previous layer and output the processing result to the next layer. The processing content of each layer includes convolution processing and activation function processing, as shown in Figure 9.

[0033] FIG. 7 is a diagram showing an example of software for processing one layer of the neural network processing of FIG.

[0034] This processing is executed by the GPU 511. I8 , k I8 , out I8 is an 8-bit integer, d I32 is a 32-bit integer, d F , p F , q F , out F , r F , s F is a floating-point number. H indicates the number of pixels in the vertical direction of the image, and loop processing using h indicates processing for each vertical coordinate of the image. W indicates the number of pixels in the horizontal direction of the image, and loop processing using w indicates processing for each horizontal coordinate of the image. CO indicates the number of types of attribute information for each pixel of the output data, and loop processing using co indicates processing for each type of attribute information for a single pixel of the output data. The value of CO is often in the tens to hundreds. Inside the loop using co, convolution (see symbol 701), FP conversion (floating-point conversion) (see symbol 702), activation function (see symbol 703), and integer conversion (see symbol 704) are processed. Convolution is performed using the input (in I8 ) and weights (k I8 ) is a multiplication and addition operation. Because the number of bits in the calculation result is larger than the input and weights, the calculation result (d I32) is a 32-bit integer. CI indicates the number of types of attribute information for each pixel of the input data, and the loop processing by ci indicates processing for each type of attribute information for a single pixel of the input data. The value of CI is usually 3 (corresponding to the three primary colors R (red), G (green), and B (blue)) for layer 1, which uses the input image as input data, but for other layers, the output of the previous layer is used as input, so it is often tens to hundreds. The activation function is the result of the FP processing (d F ) and calculate a predetermined function (f()). The details of the FP conversion and integer conversion processes will be described later.

[0035] FIG. 8 is a diagram showing the contents of the integer conversion process in FIG.

[0036] The minimum value of the data to be converted to integer is XF min , the maximum value is XF max In this case, XF min is the minimum value of an 8-bit integer (XI8 min =-128), XF max is the maximum value of an 8-bit integer (XI8 max =127) is approximated by a straight line (linear equation). min , XF max The value of is obtained by running the neural network processing before integerization on known inputs. The value when running on unknown inputs is XF min and XF max XF min XI8 for smaller inputs min , XF max XI8 for larger inputs max Although it is preferable to convert the result of calculation of a linear expression to an integer, this is omitted in Figures 7 and 8. Although it is not clearly stated in Figure 8, when converting the result of calculation of a linear expression to an integer, the largest integer that does not exceed the calculation result is selected. For example, if the result of calculation of a linear expression is -127.9, the result of integer conversion is -128. When writing a program in C language, simple integer conversion is interpreted as a process of discarding decimal points, so when -127.9 is converted to an integer, it becomes -127, which is not the expected result. Therefore, in Figure 7, the calculation result of the linear expression (r F *out F+s F ) and convert it to an integer in C (int()), then subtract 128 to obtain the integerized result. By using this method, for example, if the calculation result of a linear expression is -127.9, adding 128 will give 0.1, converting it to an integer in C (int()) will give 0, and subtracting 128 will give -128, so you can get the expected result. XF min , XF max Since the value of differs for each layer, the coefficient of the linear equation used for integerization (r in Figure 7) F , s F ) differs for each layer. The floating-point conversion process is the inverse function of integer conversion, so it can be calculated using a linear equation, just like integer conversion. The floating-point conversion in Figure 7 targets the results of multiply-and-accumulate operations on the input and weight, so the floating-point conversion coefficients are calculated backward from the coefficients when the input and weight are converted to integers.

[0037] Figure 9 is a diagram showing the process flow when the software in Figure 7 is executed, and the exchange of data with the memory. In Figure 9, "R" means reading from RAM 510_2, "C" means convolution, "F" means FP (=floating point) conversion, "A" means activation function processing, "I" means integer conversion, and "W" means writing to memory.

[0038] The neural network processing device 508 reads the calculation result (output) of the layer 901 (e.g., layer n-1 (901_n-1)) immediately before the target layer (e.g., layer n (901_n)) from the RAM 510_2, and performs convolution in the target layer using the integer as the calculation result. The neural network processing device 508 converts the integer as the convolution result into a floating-point number, and performs activation function processing using the floating-point number obtained by the FP conversion. The neural network processing device 508 converts the floating-point number obtained by the activation function processing into an integer, and writes the integer obtained by the integer conversion processing into the RAM 510_2 as the calculation result of the target layer.

[0039] In this embodiment, the activation function processing is a processing using floating-point numbers, and processing using integers cannot be performed. Therefore, the output of the activation function processing in the layer is a floating-point number (output (FP)), but integer conversion processing is performed after the activation function processing. Therefore, the integer conversion result is written to the RAM 510_2. Therefore, the data stored in the RAM 510_2 becomes an 8-bit integer (layer n-1 output (integer), layer n output (integer)), and the size of the data stored in the memory is reduced compared to FIG. 4. Then, the data stored in the RAM 510_2 (8-bit integer) is read out and used for the convolution processing, so the convolution processing can be processing using 8-bit integers, and the convolution processing time is also reduced. As a result, the processing time of the neural network processing device can be reduced overall.

[0040] Note that the integers stored in the RAM 510_2 as the calculation results for each layer 901 may be accumulated in the RAM 510_2 as a log of the calculation results. Since the integers for each layer 901 are recorded in the RAM 510_2, it is inferred that the neural network processing according to the first embodiment is being performed. [Example]

[0041] A description will be given of Example 2. In doing so, differences from the previous example will be mainly described, and explanations of commonalities with the previous example will be omitted or simplified (this also applies to Example 2 and subsequent examples).

[0042] FIG. 10 is a diagram showing a second example of the configuration of the neural network processing executed by the neural network processing device of FIG.

[0043] The configuration in Fig. 10 is characterized in that the processing of layer 1001 is not serial, and the input of a certain layer is the output of multiple layers preceding that certain layer. Specifically, for example, the result of performing addition processing 1003 on the output of layer-i (1001_i) and the output of layer-j (1001_j) next to layer-i (1001_i) is the input of layer-k (1001_k) next to layer-j (1001_j). Because the coefficients of the linear expression used to convert to integers are different between layer-i (1001_i) and layer-j (1001_j), the addition processing 1003 cannot be performed as integers and must be converted to floating point before being performed.

[0044] FIG. 11 is a diagram showing an example of software for executing the addition process 1003 of the neural network process in FIG.

[0045] The GPU running the software receives the first input (in_i I8 ) into floating point (see symbol 1101_i) and the second input (in_j I8 ) is converted to a floating-point number (see reference numeral 1101_j), then added (see reference numeral 1102), the result is converted to an integer (see reference numeral 1103_k), and the integerized result is output (out_k I8 ) The first input (in_i I8 ) to floating point coefficient (p_i F , q_i F ) uses the value calculated in advance from the coefficient when converting the output to an integer in layer i (1001_i). Similarly, the second input (in_j I8 ) to floating point coefficient (p_j F , q_j F ) uses the value calculated in advance from the coefficient when converting the output to an integer in layer j. The coefficient when converting the output to an integer (r_k F , s_k F ) uses values ​​calculated in advance from the minimum and maximum values ​​of the addition results when the neural network processing before integer conversion is performed with known inputs.

[0046] According to the second embodiment, it is possible to prevent the output of the addition process 1003 from being the sum of an integer as the output of the first layer and an integer as the output of the second layer, and from falling below the minimum value of the integer or exceeding the maximum value of the integer. Note that the integer as the output of the layer-i (1001_i) input to the addition process 1003 (first input (in_i I8 )) is also an integer (second input (in_j)) as the output of layer-j (1001_j) input to the addition process 1003. I8 )) may be integers read out from the RAM 510_2. I8 )) may be written to RAM 510_2 and read from RAM 510_2 during processing of layer-k (1001_k). [Example]

[0047] FIG. 12 is a diagram showing a third example of the configuration of the neural network processing executed by the neural network processing device of FIG.

[0048] The configuration of FIG. 12 is characterized in that the processing of each layer 1201 includes bias addition.

[0049] FIG. 13 is a diagram showing an example of software for processing one layer of the neural network processing of FIG.

[0050] In the bias value correction step (see reference numeral 1301), the bias value (b F [co]) to the floating-point coefficient (q F) is added. This makes it possible to perform coefficient addition and bias addition during floating-point conversion in a single addition process, thereby reducing processing time compared to performing floating-point conversion and bias addition separately. Specifically, in FIG. 13, reference numeral 1302 represents convolution, reference numeral 1303 represents FP conversion and bias addition, reference numeral 1304 represents activation function processing, and reference numeral 1305 represents integer conversion processing. One loop process (one layer) includes an H loop (a loop for the number of vertical pixels in the image), a W loop (a loop for the number of horizontal pixels in the image), and a CO loop (a loop for the number of types of attribute information for each pixel of the output data), but bias value correction is performed outside of these loops (before they start). This eliminates the need for bias value correction within the loops, thereby reducing the overall processing load.

[0051] For example, a neural network that is a combination of the third embodiment and the second embodiment may be adopted. That is, a layer other than the addition process (addition layer) may include bias addition. [Example]

[0052] In the above-described embodiment, the convolution process and the activation function process are executed in one loop (one layer). Therefore, if there are multiple variations of each of the convolution process and the activation function process, it is necessary to write a number of processes equivalent to the product of the number of variations of both the convolution process and the activation function process. This may require a huge amount of time to write the software and check its operation.

[0053] FIG. 14 is a diagram showing examples of variations in software for performing product-sum calculation processing and corresponding parameters.

[0054] 14, 16, and 17, "input" represents the input format, "weight" represents the weight format, "output" represents the output format, "size" represents the weight size, "CI" represents the number of types of attribute information for each pixel in the input data, and "CO" represents the number of types of attribute information for each pixel in the output data.

[0055] The input format indicates the order in which the input data is arranged, and there are two types: CHW and HWC. CHW first arranges the data along the axis of attribute information type, then arranges data along the vertical axis of the image with data that has the same values ​​on the attribute information type axis, and finally arranges data along the horizontal axis of the image with data that has the same values ​​on the attribute information type axis and the vertical axis of the image. HWC first arranges the data along the vertical axis of the image, then arranges data along the horizontal axis of the image with data that has the same values ​​on the vertical axis of the image, and finally arranges data along the axis of attribute information type with data that has the same values ​​on the vertical and horizontal axes of the image. Note that the software shown in Figures 7, 11, and 13 is software that supports the HWC input format and cannot be used when the input format is CHW.

[0056] The weight format indicates the sorting order of the weight data and has two types: CHW and HWC. The meaning of CHW and HWC is the same as that of the input format. The software shown in Figures 7, 11, and 13 is software that supports the HWC weight format and cannot be used when the weight format is CHW.

[0057] The output format indicates the sorting order of the output data and has two types: CHW and HWC. The meanings of CHW and HWC are the same as those of the input format. The software shown in Figures 7, 11, and 13 is software that supports the HWC output format and cannot be used when the output format is CHW.

[0058] The weight size indicates the range of input data pixels that affect each pixel of the output data. In the software shown in Figures 7 and 13, each pixel of the output data is only influenced by pixels in the input data that have the same values ​​on the vertical and horizontal axes. This is represented as 1x1 in Figure 14. There is also a calculation method in which each pixel of the output data is influenced by pixels with the same values ​​on the vertical and horizontal axes, as well as a 3-pixel square of input data that is expanded by one pixel above and below, and this is represented as 3x3 in Figure 14.

[0059] As mentioned above, CI is the number of types of attribute information for each pixel of the input data, and as described in the explanation of Figure 7, it can take on a value of 3 for layer 1 and tens to hundreds for other layers. Here, only two values, 3 and 32, are shown due to space constraints.

[0060] As described above, CO is the number of types of attribute information for each pixel of the output data, and can take on values ​​ranging from tens to hundreds, as described in the explanation of Fig. 7. Here, only one type, 32, is shown due to space limitations.

[0061] Although Figure 14 does not cover all the variations, there are still 32 variations, which is a lot.

[0062] Figure 15 shows variations in activation function processing. This figure lists the names of each activation function, but omits the function formulas. While Figure 15 does not cover all possible variations, there are still 10 variations.

[0063] Combining the variations in Figures 14 and 15 results in 320 combinations. In other words, it will be necessary to describe 320 different processes. For this reason, it is estimated that creating the software description and verifying its operation will take an enormous amount of time. To address this issue, a method can be considered in which processing is described only for the combinations that are required depending on the application. The development flow for this is explained below.

[0064] Figure 16 is a diagram showing an example of a combination of convolution processing parameters and activation function processing for each layer of the neural network processing in Figure 6. When performing complex recognition processing, it is often the case that 100 or more layers are executed, but for space reasons, an example with 30 layers is shown here. As can be seen from Figure 16, there are many layers with the same parameter sets.

[0065] Figure 17 shows the results of selecting combinations of parameters and activation function processing, excluding overlaps, from the combinations of parameters and activation function processing in Figure 16. As a result, it can be confirmed that there are only three combinations of parameters and activation function processing, and therefore it is sufficient to write three different software.

[0066] If a GPU manufacturer were to create a library, it would be necessary to create 320 different descriptions to accommodate all of the convolution and activation function processing variations, since it is unclear which variations the user will use. Such a large number of variations could become a bottleneck, making it difficult to create a library. From the user's perspective, however, they only need to describe the processing for the combinations required for their application. Therefore, if the software is created according to Figure 17, it is possible to shorten (e.g., significantly shorten) the time required to create the software description.

[0067] Specifically, for example, the development flow may be as follows: For each layer, a parameter set including convolution processing parameters (e.g., input format, weight format, output format, size, CI, and CO) and activation function parameters is specified by the user. This results in a list of parameter sets. Next, parameter sets are aggregated by selecting parameter sets from the specified parameter sets (parameter set list) with duplicates removed. Software is written based on the aggregated parameter sets. This development flow can reduce the software design time. Note that this software design may be performed by a neural network processing device, or may be performed by a device separate from the neural network processing device (e.g., a computer such as a personal computer). [Example]

[0068] Another way to reduce the time required to write software is to focus on the applications that neural network processing is used in. Neural network processing is generally used for two types of applications: learning processing and inference processing.

[0069] In the training process, known data is input as input to the neural network processing, and weight data is adjusted so that the output of the neural network processing approaches an expected value. Specifically, for example, the neural network is trained using a training data set that includes known images and correct labels of the known images.

[0070] On the other hand, inference processing uses unknown data as input to neural network processing and the output of the neural network processing as the recognition result.

[0071] The learning process is usually performed on a server (an example of a core device) with abundant computational resources, and there are fewer restrictions on the processing time. Therefore, there is relatively little need to reduce the processing time.

[0072] On the other hand, there are cases where computational resources are limited, such as when inference processing is performed on equipment (an example of an edge device) mounted on a train or automobile. Furthermore, when recognition results are used to control equipment, such as in autonomous driving, processing time constraints are relatively strict in order to ensure timely control of the equipment. Therefore, there is a relatively high need to reduce processing time.

[0073] Therefore, by applying software that writes integers obtained by integer conversion processing in accordance with the present invention to memory as output for the software used in the inference processing, and by using software designed without applying the present invention for the software used in the learning processing (for example, software that performs all processing using floating-point numbers), it is possible to shorten the time required to write the software while meeting the needs.

[0074] Although several embodiments have been described above, these are merely examples for the purpose of explaining the present invention, and the scope of the present invention is not limited to these embodiments. The present invention can be implemented in various other forms. For example, the above embodiments can be summarized as follows. The following summary may include a supplementary explanation of at least one embodiment, or may include a description of a modified example.

[0075] According to a first aspect, a neural network processing device (e.g., 508) includes an interface device (e.g., a plurality of interface circuits 503_1 to 503_5), a memory (e.g., RAM 510_2), and a processor (e.g., GPU 511) connected to the interface device and the memory, and performs neural network processing of data (e.g., image data from a camera 513) input via the interface device. In each of a plurality of layers in the neural network, the processor performs convolution processing to perform product-sum operations between the integers input to the layer and weight data, performs floating-point conversion processing to convert the integers output from the convolution processing into floating-point numbers, performs activation function processing to calculate a predetermined function using the floating-point numbers output from the floating-point conversion processing, performs integer conversion processing to convert the floating-point numbers output from the activation function processing into integers, and performs write processing to write the integers obtained by the integer conversion processing into memory. As a result, the data stored in the memory is converted into integers, thereby reducing the amount of data. That is, it is possible to reduce the amount of data transfer between the arithmetic unit (for example, the GPU 511) and a memory (for example, the RAM 510_2) connected to the outside of the arithmetic unit, thereby reducing the processing time of the neural network processing device.

[0076] The "predetermined function" may be a predetermined activation function, specifically an activation function that uses floating-point numbers (an activation function that cannot be processed using integers).

[0077] Furthermore, the data sizes of floating-point numbers and integers depend on the hardware, such as a GPU, that executes the software for neural network processing. However, in general, hardware that executes software for neural network processing often supports integers with a smaller data size than floating-point numbers in order to achieve both accuracy and processing speed (e.g., to achieve high-speed processing with the desired accuracy). For example, when floating-point numbers are 16 bits or 32 bits, 8-bit integers are often supported. When the data size of floating-point numbers is 8 bits, 4-bit integers are often supported. Software for neural network processing that takes into account such hardware characteristics is executed by a neural network processing device.

[0078] Furthermore, the "multiple layers" in a neural network may refer to all or some of the layers that make up the neural network. For example, for the first layer, the integers read as data from memory may be integers obtained by performing integer processing on input data to the neural network and stored in memory. Furthermore, for each of the "multiple layers," the integers input to that layer may be integers written in a previous process and read from memory in the processing of that layer. This "previous process" may be processing of a previous layer (e.g., the layer immediately preceding) or the addition process described above.

[0079] According to the second aspect, for each of the "plurality of layers" in the first aspect, integers stored in memory as processing results for that layer may be accumulated in memory as a processing result log. Since the integers for each layer are recorded in memory, it is possible to infer that neural network processing according to the present invention has been performed.

[0080] According to a third aspect, the neural network of the first or second aspect may include one or more addition layers in addition to the "plurality of layers" (plurality of convolutional layers). Each of the one or more addition layers is a processing layer that adds outputs from some of the plurality of convolutional layers and outputs the result to one of the plurality of convolutional layers. In each of the one or more addition layers, the processor may convert each of the plurality of integers input to the addition layer into a floating-point number, add the plurality of floating-point numbers, convert the resulting floating-point number into an integer, and write the converted integer to memory. This makes it possible to prevent the output of the addition layer from falling below the minimum integer value or exceeding the maximum integer value.

[0081] According to a fourth aspect, in any one of the first to third aspects, in each of the "plural layers" (plurality of convolutional layers), the processor performs a bias addition process that adds a bias value to the output of the floating-point conversion process, and the predetermined function in the activation function process may be a function calculated from the output of the bias addition process, and the processor may perform the addition process and the bias addition process included in the floating-point conversion process simultaneously. This reduces the processing load for each layer as a whole.

[0082] According to the fifth aspect, in the fourth aspect, specifically, in each of a plurality of layers, the layer may include a loop process, and the loop process may include a convolution process, a floating-point conversion process, an activation function process, and an integer conversion process, and the processor may perform a bias value correction by adding a floating-point conversion coefficient to a bias value before starting the loop process, thereby simultaneously performing an addition process and a bias addition process included in the floating-point conversion process in the loop process.

[0083] According to a sixth aspect, in any one of the first to fifth aspects, the neural network processing software may be first software written based on parameter sets obtained by excluding duplicates from a list of parameter sets including parameters for convolution processing and parameters for activation function processing for a plurality of layers, thereby reducing the time required to write the software.

[0084] According to a seventh aspect, in any one of the first to sixth aspects, of a learning process in which known data is input to a neural network and weight data (each weight in the neural network) is adjusted so that the output of the neural network approaches an expected value, and an inference process in which unknown data is input to the neural network and the output of the neural network is used as a recognition result, the software for the inference process may be the software of the first aspect. A device on which the inference process is executed is generally a so-called edge device (a device on the edge side), and therefore has fewer computational resources than a core device (a device on the core side). However, even with such computational resources, software that can perform neural network processing as the inference process can be created in a short time.

[0085] In consideration of the explanations of the fourth and fifth embodiments and the first software described above, there is, for example, a software creation method expressed as follows: The following software creation method is executed by a computer. <Expression 1> A method for creating software for neural network processing that combines multiple layer processes, including convolution processing that performs product-sum operations between input data and weight data and activation function processing that calculates a predetermined function from the results of the convolution processing, Listing parameters of convolutional processes and activation function processes of the plurality of layer processes; selecting a set of parameters from the list of parameters with duplicates removed; a step of writing software to execute the convolution process with integers, execute the activation function process with floating-point numbers, and convert the result of the activation function process into integers and write the integers to memory for the layer process corresponding to the set of parameters; A method for creating software for neural network processing, comprising: <Expression 2> A method for creating software for neural network processing that combines multiple layer processes, including convolution processing that performs product-sum operations between input data and weight data and activation function processing that calculates a predetermined function from the results of the convolution processing, The learning processing software, which inputs known data into the neural network processing and adjusts the weight data so that the output of the neural network processing approaches an expected value, is written to execute the convolution processing using floating-point numbers; The software for the inference process, which inputs unknown data to the neural network process and uses the output of the neural network process as a recognition result, is written so that the convolution process is performed using integers, the activation function process is performed using floating-point numbers, and the processing result of the activation function process is converted into an integer and written to memory. A method for creating software for neural network processing, characterized by: [Explanation of symbols]

[0086] 502 CPU 511 GPU 508 Neural network processing unit 509 ROM 510 RAM

Claims

1. A neural network processing device comprising an interface device, a memory, and a processor connected to the interface device and the memory, wherein the processor executes software for neural network processing of data input via the interface device to perform the neural network processing, The neural network processing includes, in each of a plurality of layers in a neural network, A convolution process that performs a multiply-and-accumulate operation between the integers input to the layer and the weight data; a floating-point conversion process for converting the integer output of the convolution process into a floating-point number; an activation function process that calculates a predetermined function using floating-point numbers as the output of the floating-point conversion process; an integerization process for converting a floating-point number output from the activation function process into an integer; a write process of writing the integer obtained by the integer conversion process into a memory; Including, A neural network processing device comprising:

2. For each of the plurality of layers, integers stored in the memory as processing results for that layer are accumulated in the memory as a processing result log.

2. The neural network processing device according to claim 1.

3. The neural network includes one or more addition layers in addition to the plurality of convolution layers, which are the plurality of layers; each of the one or more addition layers is a processing layer that adds outputs of some layers among the plurality of convolution layers and outputs the sum to one of the plurality of convolution layers; The neural network processing includes, in each of the one or more summation layers: converting each of the plurality of integers input to the addition layer into a floating-point number, adding the plurality of floating-point numbers, and then converting the resulting floating-point number into an integer, and writing the converted integer into the memory; Including, 2. The neural network processing device according to claim 1.

4. In each of the plurality of layers, the neural network processing includes a bias addition process of adding a bias value to an output of the floating-point conversion process; the predetermined function in the activation function processing is a function calculated from the output of the bias addition processing, the neural network processing includes simultaneously performing an addition process included in the floating-point conversion process and the bias addition process; 2. The neural network processing device according to claim 1.

5. In each of the plurality of layers, The layer includes a loop process, the loop processing includes the convolution processing, the floating-point conversion processing, the activation function processing, and the integer conversion processing, the neural network processing includes performing bias value correction by adding a coefficient for floating-point conversion to a bias value before starting the loop processing, thereby simultaneously performing the addition processing included in the floating-point conversion processing and the bias addition processing in the loop processing; 5. The neural network processing device according to claim 4.

6. the software is first software written based on parameter sets obtained by excluding duplicates from a list of parameter sets including parameters for the convolution processing and parameters for the activation function processing for the plurality of layers; 2. The neural network processing device according to claim 1.

7. software for the inference processing of the neural network learning processing and the neural network inference processing is the first software; 7. The neural network processing device according to claim 6.

8. In each of the multiple layers in the neural network, A convolution process that performs a multiply-and-accumulate operation between the integers input to the layer and the weight data; a floating-point conversion process for converting the integer output of the convolution process into a floating-point number; an activation function process that calculates a predetermined function using floating-point numbers as the output of the floating-point conversion process; an integerization process for converting a floating-point number output from the activation function process into an integer; a write process of writing the integer obtained by the integer conversion process into a memory; A neural network processing method characterized by carrying out the above by a computer.

Images