Information processing device, information processing method, information processing program, software creation device, software creation method, and software creation program

Abstract

To reduce the time of processing for neural network processing.SOLUTION: An information processor has a processor and a memory. The processor executes neural network processing including convolution processing of generating plural pieces of intermediate data by performing a product-sum operation on between plural pieces of input data and plural pieces of weight data and activation function processing of generating plural pieces of output data by calculating the value of a predetermined function with the plural pieces of intermediate data as input. The processor executes the activation function processing in loop processing of the convolution processing.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing program, a software creation device, a software creation method, and a software creation program. [Background technology]

[0002] A first conventional example of a neural network processing device that processes information using a neural network that mimics the structure of neurons in the human brain is described in Patent Document 1. This neural network processing device recognizes images, etc., by combining multiple layers that perform convolution processing, which performs product-sum operations between multiple input data and multiple weight data to generate multiple intermediate data, and activation function processing, which uses the multiple intermediate data as input and calculates a predetermined function to generate multiple output data. This conventional example also describes the use of a circuit called a GPU (Graphics Processing Unit), which has many built-in arithmetic units, to perform the convolution processing at high speed.

[0003] A second conventional example of a neural network processing device is described in Patent Document 2. This neural network processing device is capable of high-speed processing by providing a dedicated inference calculation unit having a matrix calculation unit and an activation function calculation unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-87072 [Patent Document 2] Japanese Patent Publication No. 2020-112901 Summary of the Invention [Problem to be solved by the invention]

[0005] Neural network processing devices are increasingly being used in environments with limited computing resources, such as in automated driving systems for automobiles, where they perform recognition processing on surrounding images captured by cameras. Furthermore, automated driving systems for automobiles require recognition processing over a wide range of surroundings, and high recognition accuracy is required because an error in the recognition result could directly lead to an accident. Given this background, neural network processing has become complex, and the amount of computation required to perform recognition processing on a large number of images is enormous. Processing a huge amount of computation with limited computing resources results in longer than expected processing time, resulting in the problem of not being able to achieve the required processing performance.

[0006] While the first conventional example described above shortens processing time by using a GPU, the effect of shortening processing time when processing using software on general-purpose hardware is limited, and in many cases the required processing performance cannot be achieved. The second conventional technology described above uses a dedicated inference calculation unit, which may shorten processing time compared to using a GPU. However, developing a dedicated inference calculation unit requires enormous development costs, making it difficult to apply to devices with low shipping volumes. Therefore, there is a need to shorten processing time when performing neural network processing using software on general-purpose hardware.

[0007] In order to deal with such a situation, an object of the present invention is to shorten the processing time of neural network processing. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the information processing device of the present invention is an information processing device having a processor and a memory, wherein the processor executes neural network processing including a convolution process that performs a product-sum operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data, and an activation function process that calculates the value of a predetermined function that uses the plurality of intermediate data as input to generate a plurality of output data, and the activation function process is executed within a loop process of the convolution process. [Effects of the Invention]

[0009] According to the present invention, the volume of intermediate data is reduced, making it possible to store the intermediate data inside the processing unit, and data transfer between the processing unit and external memory is reduced, thereby shortening the processing time of neural network processing. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a neural network processing device according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating a configuration of neural network processing. [Figure 3] FIG. 1 is a diagram showing an algorithm for processing one layer of neural network processing according to a first conventional technique. [Figure 4] FIG. 10 is a diagram showing a processing mode during execution of a neural network processing algorithm according to the first prior art. [Figure 5] FIG. 2 is a diagram showing an algorithm for processing one layer of neural network processing according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing a processing mode during execution of a neural network processing algorithm according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing an algorithm for processing one layer of neural network processing according to the second prior art. [Figure 8] FIG. 10 is a diagram showing a processing mode during execution of a neural network processing algorithm according to a second conventional technique. [Figure 9] FIG. 10 is a diagram showing an algorithm for processing one layer of neural network processing according to the second embodiment. [Figure 10] FIG. 10 is a diagram showing a processing mode during execution of a neural network processing algorithm according to the second embodiment. [Figure 11]FIG. 11 is a diagram showing an algorithm for processing one layer of neural network processing according to the third embodiment. [Figure 12] FIG. 11 is a diagram showing a processing mode during execution of a neural network processing algorithm according to the third embodiment. [Figure 13] 13 is a diagram showing examples of variations of an algorithm for performing product-sum operation processing according to the fourth embodiment and corresponding parameters. FIG. [Figure 14] FIG. 11 is a diagram showing examples of variations of an algorithm for performing activation function processing according to the fourth embodiment and corresponding parameters. [Figure 15] FIG. 11 is a diagram illustrating an example of a combination of parameters for product-sum operation processing and activation function processing for each layer of neural network processing according to the fourth embodiment. [Figure 16] FIG. 11 is a diagram showing the results of selecting parameter combinations in which overlapping has been removed from the parameter combinations according to the fourth embodiment. [Figure 17] FIG. 10 is a diagram showing an example of hardware of a computer that executes a software creation process according to a fourth embodiment. [Figure 18] 10 is a flowchart showing a software creation process according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings. The following embodiments, including the drawings, are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0012] In the following embodiments, processing executed by a program may be described. A computer such as a server or a client executes a program using a processor (e.g., a CPU (Central Processing Unit), a GPU), and performs processing steps defined in the program using storage resources (e.g., memory), interface devices (e.g., communication ports), etc. Therefore, the entity that executes the program and performs the processing may be the processor. Similarly, the entity that executes the program and performs the processing may be a controller, device, system, computer, or node that has a processor. The entity that executes the program and performs the processing may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a CPLD (Complex Programmable Logic Device), a quantum computer, etc.

[0013] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0014] In the following embodiments, a "YYY (processing) unit," a "YYY step," and a "YYY procedure" are corresponding components of different categories of devices (or systems), methods, and programs. For example, a processor or computer that executes a "YYY step" corresponds to a "YYY processing unit." Furthermore, a "YYY procedure" executed by a processor or computer corresponds to a "YYY step."

[0015] In the following embodiments, when there are multiple components having the same or similar functions, they may be described by using the same reference numerals with different subscripts. Also, when there is no need to distinguish between these multiple components, the subscripts may be omitted.

[0016] In the following embodiments, "software" may be a component including programs and other files such as libraries, or may be the program itself.

[0017] [Embodiment 1] (Configuration of information processing device 10 according to embodiment 1) 1 is a diagram showing the configuration of an information processing device 10 according to embodiment 1. The information processing device 10 includes a microprocessor 11, a ROM (Read Only Memory) 12, RAMs (Random Access Memories) 13 and 15, and a GPU 14.

[0018] The microprocessor 11 includes a CPU 11a and interface circuits 11b1, 11b2, 11b3, 11b4, and 11b5.

[0019] The ROM 12 is connected to the microprocessor 11 via an interface circuit 11b1. The RAM 13 is connected to the microprocessor 11 via an interface circuit 11b2. The GPU 14 is connected to the RAM 15. The GPU 14 is also connected to the microprocessor 11 via an interface circuit 11b3.

[0020] The cameras 111a and 111b are connected to the microprocessor 11 via an interface circuit 11b4. The display device 112 is connected to the microprocessor 11 via an interface circuit 11b5.

[0021] The information processing device 10 receives image data from the cameras 111a and 111b, performs recognition using neural network processing, and then displays the recognition result on the display device 112. The recognition result may also be output as sound from a speaker. Note that, although this embodiment shows an example of performing image recognition, the present invention is not limited to image recognition and can be applied to other general neural network processing such as sound recognition.

[0022] The cameras 111a and 111b each capture images in different directions. This makes it possible to capture all images of the area to be recognized and perform recognition processing. Cameras with a large number of pixels and a wide shooting range may also be used as the cameras 111a and 111b. This makes it possible to cover the entire area with a small number of cameras, even if the area to be recognized is large. In this case, an image captured by one camera may be divided into multiple areas and recognition processing may be performed separately. This makes it possible to exclude areas that do not need to be recognized, such as the sky, and perform recognition processing only on the necessary areas.

[0023] The microprocessor 11 is an LSI (Large Scale Integration) in which the CPU 11a and interface circuits 11b1, 11b2, 11b3, 11b4, and 11b5 are integrated on a single chip. This configuration is an example, and some or all of the ROM 12, RAM 13, GPU 14, and RAM 15 may be built into the microprocessor 11.

[0024] The CPU 11a reads and executes software stored in the ROM 12 via the interface circuit 11b1. Note that since the read / write speed of the ROM 12 is often slower than that of the RAM 13, the software may be copied from the ROM 12 to the RAM 13 at startup, and then the software may be read from the RAM 13. The CPU 11a performs the following series of processes in accordance with the software read from the ROM 12 or RAM 13.

[0025] First, the CPU 11a acquires image data from the cameras 111a and 111b via the interface circuit 11b4, and stores the image data in the RAM 13 via the interface circuit 11b2. Next, the CPU 11a reads the image data stored in the RAM 13 via the interface circuit 11b2, and transfers the image data to the GPU 14 via the interface circuit 11b3.

[0026] Next, the CPU 11a reads out software stored in the ROM 12 via the interface circuit 11b1 or in the RAM 13 via the interface circuit 11b2, and transfers it to the GPU 14 via the interface circuit 11b3. The CPU 11a then instructs the GPU 14 to start calculations using the software.

[0027] Next, when the CPU 11a receives a notice of the end of the calculation from the GPU 14 via the interface circuit 11b3, it acquires the calculation result from the GPU 14 via the interface circuit 11b3 and stores it in the RAM 13 via the interface circuit 11b2.

[0028] Next, the CPU 11a reads the calculation results of the GPU 14 from the RAM 13 via the interface circuit 11b2, performs predetermined processing, and then displays the results on the display device 112 via the interface circuit 11b5.

[0029] When the GPU 14 receives image data from the CPU 11a via the interface circuit 11b3, it stores the image data in the RAM 15. When the GPU 14 receives software from the CPU 11a via the interface circuit 11b3, it executes the software and stores the calculation results in the RAM 15.

[0030] Furthermore, when the GPU 14 receives a request to read out the calculation result from the CPU 11a via the interface circuit 11b3, the GPU 14 reads out the calculation result from the RAM 15 and outputs it to the CPU 11a via the interface circuit 11b3.

[0031] (Neural network processing) Fig. 2 is a diagram showing the configuration of neural network processing. Note that Fig. 2 describes the recognition processing for a single input image, and when performing recognition processing for multiple images, the processing in Fig. 2 is executed for each input image. In this case, the processing content may be the same for all images, or may be different for each image according to the purpose of the recognition processing.

[0032] The neural network processing shown in Figure 2 is composed of N layers, Layer i (Layer 1, Layer 2, ..., Layer N). Layer 1 receives an input image and outputs the processing result to Layer 2. Layer (i+1) (i = 1, 2, ..., N-2) receives data from the previous Layer i and outputs the processing result to Layer (i+2). Layer N receives data from the previous Layer (N-1) and outputs the processing result as the recognition result of the input image.

[0033] Each of the N layers i consists of convolution processing and activation function processing. Details of convolution processing and activation function processing will be described later.

[0034] (Layer processing of neural network processing according to the first conventional technique, which is a comparative example) Here, we will explain neural network processing according to a first conventional technology, which is a comparative example of embodiment 1. Figure 3 is a diagram showing an algorithm for processing one layer of the neural network processing according to the first conventional technology. The hardware that executes the neural network processing of the first conventional technology is the same as that of the information processing device 10 of embodiment 1.

[0035] When executing the layer processing (layer_n()) of the nth layer (n=1, 2, . . . , N) shown in FIG. 3, the CPU 11a starts the convolution processing (convolution_n()) on lines 7 to 20 and the activation function processing (activation_n()) on lines 22 to 32. The convolution processing (convolution_n()) performs a product-sum operation between the input data (in[]) and the weight data (k[]) to generate intermediate data (d[]). The activation function processing (activation_n()) uses the intermediate data (d[]) as input to calculate the value of a predetermined function (f()) and generate output data (out[]).

[0036] Convolution_n() and activation_n() are executed by the GPU 14. However, convolution_n() and activation_n() may also be executed by the CPU 11a.

[0037] The argument "H" of convolution_n() and activation_n() indicates the number of pixels in the vertical direction of the image. The loop processing related to the variable h on lines 9 to 19 of Figure 3 indicates processing for each coordinate in the vertical direction of the image. The argument "W" indicates the number of pixels in the horizontal direction of the image. The loop processing related to the variable "w" on lines 10 to 18 of Figure 3 indicates processing for each coordinate in the horizontal direction of the image.

[0038] The argument "CO" indicates the number of types of attribute information for each pixel of the output data. The loop processing for the variable "co" on lines 11 to 17 of Fig. 3 indicates processing for each type of attribute information for a single pixel of the output data. The value of the argument "CO" is often in the tens to hundreds.

[0039] The argument "CI" indicates the number of types of attribute information for each pixel of the input data. The loop processing for the variable "ci" on lines 13 to 16 of Figure 3 indicates processing for each type of attribute information for a single pixel of the input data. The value of the argument "CI" is generally 3 (corresponding to the three primary colors R (red), G (green), and B (blue)) for layer 1, which uses the input image as input data. For layers other than layer 1, the output of the previous layer is used as input, so the value is often in the tens to hundreds.

[0040] The argument "k" is the weight data of the layer n to be processed.

[0041] Fig. 4 is a diagram showing the processing mode during execution of the neural network processing algorithm according to the first conventional technique. As shown in Fig. 4, in the convolution process (convolution_n()), the input data (in[]) and weight data (k[]) are read from memory, and the intermediate data d[] is written to memory.

[0042] The memory includes a cache memory built into the processing unit (GPU 14 in this embodiment) and an external memory connected to the outside of the processing unit (RAM 15 in this embodiment). For example, when performing image recognition processing, the input data (in[]) and intermediate data (d[]) are large in volume and cannot be stored in the cache memory, so access to the external memory occurs.

[0043] The inventors of the present application discovered that data transfer between a processing unit and an external memory takes longer than that between a processing unit and a cache memory, and that the processing time for data transfer between the processing unit and the external memory is not negligible compared to the calculation processing itself. The same is true for activation function processing (activation_n()), and the processing time for data transfer between the processing unit and the external memory is not negligible compared to the calculation processing itself.

[0044] That is, in the first conventional technology, data transfer occurs between the GPU 14 and RAM 15 when reading input data (in[]) and writing intermediate data (d[]) for the convolution process (convolution_n()), reading intermediate data (d[]) for the activation function process (activation_n()), and writing output data (out[]), and it was found that this causes a bottleneck in neural network processing performed by software on general-purpose hardware such as the GPU 14.

[0045] (Layer processing of neural network processing according to embodiment 1) Fig. 5 is a diagram showing an algorithm for processing one layer of neural network processing according to embodiment 1. When executing the layer processing (layer_n()) shown in Fig. 5, the CPU 11a simply activates processing (convact_n()) that combines convolution processing and activation function processing, as compared to Fig. 3.

[0046] As shown in Figure 5, during the loop processing for the variable "co" on lines 10 to 16, each time the loop processing for the variable "ci" on lines 12 to 14 is completed, the activation function processing on line 15 is performed on the intermediate data (d) to generate output data (out[]).

[0047] Fig. 6 is a diagram showing a processing mode during execution of the neural network processing algorithm according to the first embodiment. The neural network processing algorithm according to the first embodiment shown in Fig. 5 differs from the algorithm of the first prior art shown in Fig. 3 in that intermediate data is stored in a one-dimensional variable d that is not an array, and therefore can be held inside the GPU 14 (for example, in a cache memory), eliminating the need for data transfer between the GPU 14 and the RAM 15. This reduces the amount of data transferred between the GPU 14 and the RAM 15, thereby shortening the processing time.

[0048] [Embodiment 2] (Layer processing of neural network processing according to the second prior art, which is a comparative example) First, we will explain neural network processing according to a second conventional technique, which is a comparative example of the second embodiment. FIG. 7 is a diagram showing an algorithm for processing one layer of neural network processing according to the second conventional technique. In the algorithm according to the first conventional technique in FIG. 3, the convolution processing is composed only of product-sum operations. On the other hand, in the algorithm according to the second conventional technique in FIG. 7, a bias value addition process on lines 28 to 38 of FIG. 7 is added to the convolution processing on lines 13 to 38. The bias value addition process is a process of adding a bias value for each layer of a neural network consisting of N layers to the result of the product-sum operation. It is known that this can improve the recognition result in some cases.

[0049] Fig. 8 is a diagram showing the processing mode during execution of the neural network processing algorithm according to the second conventional technique. Compared to Fig. 4, the processing of reading the input data (b[], d1[]) of the biased addition from RAM 15 and writing the calculation result of the biased addition (d2[]) to RAM 15 has been added, which increases the processing time for data transfer between GPU 14 and RAM 15.

[0050] (Layer processing of neural network processing according to the second embodiment) Fig. 9 is a diagram showing an algorithm for processing one layer of neural network processing according to embodiment 2. In this embodiment, the product-sum operation (multiadd_n() on lines 13 to 26 in Fig. 7), bias addition (Biasadd_n() on lines 28 to 38 in Fig. 7), and activation function processing (activation_n() on lines 40 to 50 in Fig. 7) in Fig. 7 are executed in one loop (lines 8 to 19 in Fig. 9).

[0051] As shown in Figure 9, in the loop processing for the variable "co" on lines 10 to 17, each time the loop processing for the variable "ci" on lines 12 to 14 is completed, the intermediate data (d2) is calculated by adding a bias value (b[]) to the intermediate data (d1) as on line 15. Then, as on line 16, the activation function processing on line 15 is performed on the intermediate data (d2) to generate output data (out[]).

[0052] 10 is a diagram showing a processing mode during execution of the neural network processing algorithm according to the second embodiment. The neural network processing algorithm according to the second embodiment shown in FIG. 9 differs from the algorithm of the second prior art shown in FIG. 7 in that two types of intermediate data are stored in one-dimensional variables d1 and d2, which are not arrays. This makes it possible to hold the intermediate data d1 and d2 inside the GPU 14 (for example, in a cache memory), eliminating the need for data transfer between the GPU 14 and the RAM 15. This reduces the amount of data transferred between the GPU 14 and the RAM 15, shortening the processing time.

[0053] [Embodiment 3] (Layer processing of neural network processing according to the third embodiment) Fig. 11 is a diagram showing an algorithm for processing one layer of neural network processing according to embodiment 3. As shown in Fig. 11, the algorithm for neural network processing according to embodiment 3 differs from the algorithm of embodiment 2 shown in Fig. 9 in that only the product-sum operation is executed in a different loop process (lines 8 to 19), and the bias value addition process and activation function process are executed in one loop process (lines 24 to 32).

[0054] Generally, multiply-and-accumulate operations involve a large amount of calculations, accounting for a large proportion of the processing time in overall neural network processing. Therefore, processing time is often reduced by using optimized libraries provided by GPU manufacturers, etc. In the case of the algorithm of embodiment 2 shown in Figure 9, a bias value addition process and activation function process are incorporated into the same loop process as the multiply-and-accumulate operation, so a library for the multiply-and-accumulate operation cannot be used, and the processing time for the multiply-and-accumulate operation may be longer than when a library is used. On the other hand, in the case of the algorithm of this embodiment shown in Figure 11, only the processing portion of the multiply-and-accumulate operation can be replaced with a library, preventing an increase in the processing time for the multiply-and-accumulate operation.

[0055] FIG. 12 is a diagram illustrating a processing mode during execution of an algorithm for neural network processing according to the third embodiment. Compared to the second conventional technique of FIG. 8, the neural network processing according to this embodiment eliminates the need to write and read the processing result (d2) of the bias value addition process (biasadd-n()) to and from memory, thereby reducing the amount of data transferred between the GPU 14 and RAM 15 and shortening the processing time. Furthermore, compared to the second embodiment of FIG. 10, the neural network processing according to this embodiment requires more memory write and read operations for the processing result (d1[]) of the multiply-and-accumulate operation (multiadd-n()). This increases the amount of data transferred between the GPU 14 and RAM 15 and the processing time. However, since it is possible to use a library for the multiply-and-accumulate operation (multiadd-n()), an increase in the processing time for the multiply-and-accumulate operation can be prevented, and therefore the total processing time may be reduced compared to the second embodiment of FIG. 10.

[0056] [Embodiment 4] In the embodiment disclosed herein, the convolution process and the activation function process are executed in a single loop. Therefore, if there are multiple variations of each of the convolution process and the activation function process, when creating software for neural network processing, it is necessary to write a number of processes equivalent to the product of the number of variations of both the convolution process and the activation function process, which may take an enormous amount of time to create the software and check its operation.

[0057] It is known that there are many variations in the algorithm for product-sum operation, which is a part of convolution processing. Fig. 13 is a diagram showing examples of variations in the algorithm for performing product-sum operation processing according to the fourth embodiment and corresponding parameters.

[0058] "INPUT FORM" in Fig. 13 indicates the order in which the input data is arranged, and there are two types: "CHW" and "HWC." "CHW" first arranges the data along the axis of attribute information type, then arranges data with the same values ​​on the attribute information type axis along the vertical axis of the image, and finally arranges data with the same values ​​on the attribute information type axis and the vertical axis of the image along the horizontal axis of the image.

[0059] "HWC" first arranges the data along the vertical axis of the image, then arranges the data along the horizontal axis of the image with data that has the same values ​​on the vertical axis of the image, and finally arranges the data along the axis of attribute information type with data that has the same values ​​on the vertical and horizontal axes of the image.

[0060] The algorithms shown in FIGS. 3, 5, 7, 9, and 11 are software that supports the input format "HWC" and cannot be applied when the input format is "CHW."

[0061] "WEIGHT FORM" in Figure 13 indicates the sorting order of the weight data, and there are two types: "CHW" and "HWC." The meanings of "CHW" and "HWC" are the same as those of the input format. The algorithms shown in Figures 3, 5, 7, 9, and 11 are software that supports the weight format "HWC," and cannot be applied when the weight format is "CHW."

[0062] "OUTPUT FORM" in Figure 13 indicates the sorting order of the output data, and there are two options: "CHW" and "HWC." The meanings of "CHW" and "HWC" are the same as those of the input format. The algorithms shown in Figures 3, 5, 7, 9, and 11 are software that supports the output format "HWC," and cannot be applied when the output format is "CHW."

[0063] "WEIGHT SIZE" in Figure 13 indicates the range of pixels in the input data that affect each pixel in the output data. In the algorithms shown in Figures 3, 5, 7, 9, and 11, each pixel in the output data is only affected by pixels in the input data that have the same values ​​on the vertical and horizontal axes. This is represented as "1x1" in Figure 13. There is also a calculation method in which each pixel in the output data is affected by pixels with the same values ​​on the vertical and horizontal axes, plus a 3-pixel square of input data that is expanded by one pixel above and below, and this is represented as "3x3" in Figure 13.

[0064] "CI" in Fig. 13 is the number of types of attribute information for each pixel of the input data, and as explained in Fig. 3, it can take on a value of 3 for layer 1 and tens to hundreds for other layers. Here, only two values, "3" and "32", are shown due to space limitations.

[0065] "CO" in Fig. 13 is the number of types of attribute information for each pixel of the output data, and can take on values ​​ranging from tens to hundreds, as explained in Fig. 3. Here, due to space limitations, only one type, "32", is shown.

[0066] Although FIG. 13 does not necessarily cover all variations of the algorithm for performing the sum-of-products operation, there are still 32 variations.

[0067] Fig. 14 is a diagram showing examples of variations of the activation function processing algorithm and corresponding parameters according to the fourth embodiment. Fig. 14 shows only the names of the activation functions, but does not show the calculation formulas for the functions. Although Fig. 14 does not necessarily cover all the variations, there are still 10 variations.

[0068] The number of variations in the product-sum operation process shown in Figure 13 multiplied by the variations in the activation function process shown in Figure 14 is 320. This means that in order to execute the convolution process and the activation function process in a single loop, it is necessary to write 320 different processes. For this reason, it is thought that it would take an enormous amount of time to create and check the operation of software for neural network processing that includes the product-sum operation process with the variations in Figure 13 and the activation function process with the variations in Figure 14.

[0069] To address this issue, a method can be considered in which processing is written only for the combinations that are required depending on the application. The development flow for this is explained below.

[0070] Fig. 15 is a diagram showing an example of a set of parameters for product-sum operation processing and activation function processing for each layer of neural network processing according to embodiment 4. When performing complex recognition processing, it is often the case that 100 or more layers are executed, but for space reasons, an example with 30 layers is shown here. As can be seen from Fig. 15, there are multiple layers with the same combination of parameters.

[0071] Fig. 16 is a diagram showing the result of selecting parameter combinations excluding overlapping ones from the parameter combinations according to embodiment 4. As shown in Fig. 16, as a result of selecting parameter combinations excluding overlapping ones from the parameter combinations in Fig. 15, it can be confirmed that the parameter combinations are narrowed down to three types. Therefore, it can be seen that when the convolution processing and the activation function processing are executed in one loop processing, three types of software need to be written.

[0072] When a GPU manufacturer provides a library, it is unclear which combination of convolution and activation function the user will use, and therefore it is necessary to create software written to accommodate all 320 variations. Such a large number of variations can become a bottleneck, potentially making it difficult to create a library. On the other hand, when implementing an embodiment of the present disclosure from the user's perspective, it is only necessary to write processing for only the combinations required depending on the application. Therefore, it is possible to create software by narrowing down the number of parameter combinations to, for example, three, as shown in FIG. 16, thereby significantly reducing the time required to write software.

[0073] (Hardware of the computer 100 that executes the software creation method according to the fourth embodiment) FIG. 17 is a hardware diagram showing an example of the configuration of a computer 100 that executes the software creation method according to the fourth embodiment.

[0074] The computer 100 is a computer that includes a processor 101 including a CPU, a main memory device 102, an auxiliary memory device 103, a network interface 104, an input device 105, and an output device 106, all of which are interconnected via an internal communication line 109 such as a bus.

[0075] The processor 101 controls the overall operation of the computer 100. The main memory device 102 is composed of, for example, a volatile semiconductor memory, and is used as a work memory for the processor 101. The auxiliary memory device 103 is composed of a large-capacity nonvolatile memory device such as a hard disk device, an SSD (Solid State Drive), or a flash memory, and is used to store various programs and data for a long period of time.

[0076] The software creation program 103a stored in the auxiliary memory device 103 is loaded into the main memory device 102 when the computer 100 is started up or when needed, and the processor 101 executes the software creation program 103a loaded into the main memory device 102, thereby realizing a software creation device that executes the software creation method.

[0077] The software creation program 103a may be recorded on a non-transitory recording medium, read from the non-transitory recording medium by a medium reading device, and loaded into the main memory device 102. Alternatively, the software creation program 103a may be obtained from an external computer via a network and loaded into the main memory device 102.

[0078] The network interface 104 is an interface device for connecting the computer 100 to each network within the system or for communicating with other computers. The network interface 104 is configured, for example, by a network interface card (NIC) for a wired local area network (LAN) or a wireless LAN.

[0079] The input device 105 is composed of a keyboard, a pointing device such as a mouse, and the like, and is used by the user to input various instructions and information to the computer 100. The output device 106 is composed of a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, and an audio output device such as a speaker, and is used to present necessary information to the user when necessary.

[0080] (Software Creation Process According to the Fourth Embodiment) FIG. 18 is a flowchart showing a software creation process according to the fourth embodiment.

[0081] First, in step S11, the computer 100 executes a first step of creating a list of combinations of parameters for convolution processing and activation function processing for each of a plurality of layer processings of the target neural network.

[0082] Next, in step S12, the computer 100 executes a second step of selecting a combination of parameters from the list of parameters created in step S11, excluding duplicates.

[0083] Next, in step S13, the computer 100 determines whether the software to be created is for inference processing. If the software to be created is for inference processing (step S13 YES), the computer 100 proceeds to step S14, and if the software is for learning processing (step S13 NO), the computer 100 proceeds to step S15.

[0084] In step S14, the computer 100 executes a third step of creating a program that executes the activation function processing within the loop processing of the convolution processing, for the convolution processing and activation function processing corresponding to the combination of parameters selected in step S13.

[0085] Meanwhile, in step S15, the computer 100 executes a fourth step of creating a program for executing the activation function processing in a loop process separate from the convolution processing, for the convolution processing and activation function processing corresponding to the combination of parameters selected in step S13.

[0086] Note that some or all of the steps S11 to S15 may be performed manually.

[0087] According to this embodiment, when creating software for neural network processing, it is only necessary to narrow down the combinations of parameters when creating the software, which makes it possible to significantly reduce the time required for creating the software and performing operational tests.

[0088] (Other embodiments) Another embodiment for reducing the time required to write software is to focus on the application in which neural network processing is used. Neural network processing is generally used for two types of applications: learning processing and inference processing. In learning processing, known data is input as input data, and weight data is adjusted so that output data approaches an expected value. On the other hand, inference processing, unknown data is input as input data, and output data is used as the recognition result. Learning processing is usually performed on servers with abundant computing resources, and processing time constraints are loose. Therefore, there is relatively little need to reduce processing time.

[0089] On the other hand, inference processing is sometimes performed by equipment installed in automobiles, for example, and computational resources are often limited. Furthermore, when using recognition results to control equipment, such as in autonomous driving, relatively strict processing time constraints are imposed in order to ensure timely control of the equipment. Therefore, there is a relatively high need to reduce processing time. Therefore, by applying embodiments of the present disclosure to the software used for inference processing to reduce processing time and using existing software for learning processing without applying embodiments of the present disclosure, it is possible to reduce the time required to create software while meeting needs.

[0090] In the embodiment disclosed herein, although creating software that executes activation processing within a loop of convolutional calculations for inference may be a burden on the user, the embodiment disclosed herein has the effect of achieving both high processing speed and high recognition accuracy in edge terminals such as on-board devices in automobiles, etc. In the embodiment disclosed herein, the burden on the user in creating software can be reduced by providing a program creation method that narrows down overlapping parameter combinations as in embodiment 4 and reduces the number of algorithms in the software to be created.

[0091] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, as long as there is no contradiction, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of one embodiment to the configuration of another embodiment. Furthermore, it is possible to add, delete, replace, integrate, or distribute part of the configuration of each embodiment. Furthermore, the configurations and processes described in the embodiments can be distributed, integrated, or replaced as appropriate based on processing efficiency or implementation efficiency. [Explanation of symbols]

[0092] 10: Information processing device, 100: Computer

Claims

1. An information processing device having a processor and a memory, The processor: a convolution process for generating a plurality of intermediate data by performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data; an activation function process for calculating values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; Execute software written to execute the activation function processing using the intermediate data as input each time the intermediate data is generated in the loop processing of the convolution processing.

1. An information processing device comprising:

2. 2. The information processing device according to claim 1, The processor: In the loop processing, each time the multiply-and-accumulate operation is performed to generate one piece of intermediate data among the plurality of pieces of intermediate data, the activation function processing is performed to calculate the value of the predetermined function using the one piece of intermediate data as an input, thereby generating one piece of output data among the plurality of output data.

1. An information processing device comprising:

3. 2. The information processing device according to claim 1, the convolution processing includes a bias value addition processing of adding a bias value to a result of the product-sum operation, The bias value addition process is executed in the same loop as the activation function process.

1. An information processing device comprising:

4. An information processing device having a processor and a memory, The processor: a convolution process for generating a plurality of intermediate data by performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data; an activation function process for calculating values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; the convolution processing includes a bias value addition processing of adding a bias value to a result of the product-sum operation, The processor: The bias value addition process is executed in the same loop process as the activation function process; The bias value addition process and the activation function process are executed in a loop process separate from the convolution process.

1. An information processing device comprising:

5. The information processing device according to any one of claims 1 to 4, the plurality of input data are a plurality of image data corresponding to a plurality of ranges, The output data is a result of image recognition processing on the plurality of image data.

1. An information processing device comprising:

6. An information processing method executed by an information processing device having a processor and a memory, a convolution processing step in which the processor performs a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing step in which the processor calculates values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; a neural network processing step including: The processor executes software written to execute the activation function processing step using the intermediate data as input each time the intermediate data is generated in the loop processing of the convolution processing step. An information processing method comprising:

7. An information processing method executed by an information processing device having a processor and a memory, a convolution processing step in which the processor performs a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing step in which the processor calculates values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; a neural network processing step including: the convolution processing step includes a bias value addition processing step in which the processor adds a bias value to a result of the multiply-accumulate operation, the processor: The bias value addition step is executed within the same loop processing as the activation function processing step; The bias value addition step and the activation function processing step are executed in a loop process different from the convolution processing step. An information processing method comprising:

8. A computer having a processor and a memory, a convolution processing procedure for performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing procedure that calculates values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; Executing a neural network processing procedure including Each time the intermediate data is generated in the loop processing of the convolution processing procedure, software is executed that is written to execute the activation function processing procedure using the intermediate data as input. An information processing program characterized by:

9. A computer having a processor and a memory, a convolution processing procedure for performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing procedure that calculates values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; Executing a neural network processing procedure including the convolution processing step includes a bias value addition processing step of adding a bias value to a result of the product-sum operation, the processor, Executing the bias value addition process in the same loop as the activation function process; The bias value addition process and the activation function process are executed in a loop process different from the convolution process. An information processing program characterized by:

10. A software creation device that creates software that causes an information processing device to execute neural network processing that performs recognition processing on input data by combining multiple layer processes that each include a convolution process that generates multiple intermediate data by performing a product-sum operation between multiple input data and multiple weight data, and an activation function process that uses the multiple intermediate data as input and calculates the value of a predetermined function to generate multiple output data, a first processing unit that creates a list of combinations of parameters of the convolution processing and the activation function processing for each of the layer processing; a second processing unit that selects a combination of parameters from the list, excluding duplicates; a third processing unit that creates software for executing the activation function processing within a loop process of the convolution processing, with respect to the convolution processing and the activation function processing corresponding to the combination of parameters selected by the second processing unit; A software creation device comprising:

11. 11. The software creation device according to claim 10, The third processing unit is When creating software for an inference process in which unknown data is input as the plurality of input data and the plurality of output data is used as recognition results, software is created that executes the activation function process within a loop process of the convolution process, with respect to the convolution process and the activation function process corresponding to the combination of parameters selected by the second processing unit, When creating software for a learning process in which known data is input as the plurality of input data and the plurality of weight data is adjusted so that the plurality of output data approaches expected values, software is created that executes the activation function process in a loop process separate from the convolution process, for the convolution process and the activation function process corresponding to the combination of parameters selected by the second processing unit. A software creation device characterized by:

12. A software creation method for creating software in which a computer having a processor and a memory creates software that causes an information processing device to execute neural network processing for performing recognition processing on input data by combining a plurality of layer processes each including a convolution process that performs a product-sum operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data, and an activation function process that uses the plurality of intermediate data as input and calculates the value of a predetermined function to generate a plurality of output data, the method comprising: a first step in which the computer creates a list of combinations of parameters of the convolution processing and the activation function processing for each of the layer processing; a second step in which the computer selects a combination of parameters from the list, excluding duplicates; a third step in which the computer creates software for executing the activation function processing within a loop process of the convolution processing, for the convolution processing and the activation function processing corresponding to the combination of parameters selected in the second step; A software creation method comprising:

13. A computer having a processor and a memory, A software creation program for creating software that causes an information processing device to execute neural network processing for performing recognition processing on input data by combining a plurality of layer processes each including: a convolution process that generates a plurality of intermediate data by performing a product-sum operation between a plurality of input data and a plurality of weight data; and an activation function process that uses the plurality of intermediate data as input and calculates the value of a predetermined function to generate a plurality of output data, The computer, a first step of creating a list of combinations of parameters of the convolution processing and the activation function processing for each of the layer processing; a second step of selecting a combination of parameters from the list, excluding duplicates; a third step of creating software for executing the activation function processing within a loop process of the convolution processing, for the convolution processing and the activation function processing corresponding to the combination of parameters selected in the second step; A software creation program characterized by executing the above.

14. A software creation method for creating software that causes an information processing device to execute neural network processing for performing recognition processing on input data by combining a plurality of layer processes each including a convolution process that generates a plurality of intermediate data by performing a product-sum operation between a plurality of input data and a plurality of weight data, and an activation function process that uses the plurality of intermediate data as input and calculates the value of a predetermined function to generate a plurality of output data, a first step of creating a list of combinations of parameters of the convolution process and the activation function process for each of the layer processes; a second step of selecting a combination of parameters from the list, excluding duplicates; a third step of creating software for executing the activation function processing within a loop process of the convolution processing, for the convolution processing and the activation function processing corresponding to the combination of parameters selected in the second step; A software creation method comprising:

Images