Information processor, method for processing information, information processing program, software creation device, method for creating software, and software creation program

Abstract

To reduce the time of processing for neural network processing.SOLUTION: An information processor has a processor and a memory. The processor executes neural network processing including convolution processing of generating plural pieces of intermediate data by performing a product-sum operation on between plural pieces of input data and plural pieces of weight data and activation function processing of generating plural pieces of output data by calculating the value of a predetermined function with the plural pieces of intermediate data as input. The processor executes the activation function processing in loop processing of the convolution processing.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, an information processing program, a software creation apparatus, a software creation method, and a software creation program.

Background Art

[0002] As a first conventional example of a neural network processing apparatus that performs information processing using a neural network mimicking the structure of nerve cells in the human brain, there is one described in Patent Document 1. This neural network processing apparatus performs a convolution process that calculates the sum of products between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data, and a plurality of intermediate data as inputs to calculate a predetermined function to generate a plurality of output data. A plurality of layers that execute activation function processing are combined to perform recognition of an image or the like. Further, this conventional example describes that the convolution process is executed at high speed using a circuit incorporating a large number of arithmetic units called a GPU (Graphics Processing Unit).

[0003] Further, as a second conventional example of a neural network processing apparatus, there is one described in Patent Document 2. This neural network processing apparatus enables high-speed processing by providing a dedicated inference arithmetic unit having a matrix arithmetic unit and an activation function arithmetic unit.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Neural network processors are increasingly being used in environments with limited computing resources, such as in autonomous driving systems for automobiles, where they perform image recognition on surrounding images captured by cameras. Furthermore, autonomous driving systems for automobiles require recognition processing over a wide area of ​​the surroundings, and errors in recognition results can directly lead to accidents, thus demanding high recognition accuracy. Against this backdrop, neural network processing is becoming more complex in configuration, and the need to perform recognition processing on a large number of images results in an enormous amount of computation. Processing such a large amount of computation with limited computing resources results in processing times that exceed expectations, leading to the problem of not being able to obtain the necessary processing performance.

[0006] The first conventional example above uses a GPU to reduce processing time, but the effect of reducing processing time is limited when processing software on general-purpose hardware, and in many cases the required processing performance cannot be achieved. The second conventional technology above has a dedicated inference unit, which may allow for faster processing than using a GPU, but developing a dedicated inference unit incurs enormous development costs, making it difficult to apply to devices with low shipment volumes. Therefore, there is a need to reduce processing time when performing neural network processing with software on general-purpose hardware.

[0007] To address this situation, the present invention aims to shorten the processing time of neural network processing. [Means for solving the problem]

[0008] To solve the above-mentioned problems, the present invention provides an information processing device having a processor and a memory, wherein the processor performs a neural network process that includes a convolution process which generates a plurality of intermediate data by performing a sum-of-accumulate operation between a plurality of input data and a plurality of weight data, and an activation function process which generates a plurality of output data by calculating the value of a predetermined function that takes the plurality of intermediate data as input, and the activation function process is performed within a loop process of the convolution process. [Effects of the Invention]

[0009] According to the present invention, since the capacity of intermediate data is reduced, it becomes possible to store intermediate data inside the processing unit, and since data transfer between the processing unit and external memory is reduced, the processing time for neural network processing can be shortened. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the configuration of a neural network processing device according to Embodiment 1. [Figure 2] This diagram shows the configuration of neural network processing. [Figure 3] This figure shows an algorithm for processing one layer of a neural network processing in the first prior art. [Figure 4] This figure shows the processing mode during execution of the neural network processing algorithm related to the first prior art. [Figure 5] This figure shows an algorithm for processing one layer of the neural network processing according to Embodiment 1. [Figure 6] This figure shows the processing mode during execution of the neural network processing algorithm according to Embodiment 1. [Figure 7] This figure shows an algorithm for processing one layer of a neural network processing in the second prior art. [Figure 8] This figure shows the processing mode during execution of a neural network processing algorithm related to the second prior art. [Figure 9] This figure shows an algorithm for processing one layer of the neural network processing according to Embodiment 2. [Figure 10] This figure shows the processing mode during execution of the neural network processing algorithm according to Embodiment 2. [Figure 11]It is a diagram showing an algorithm for processing one layer of neural network processing according to Embodiment 3. [Figure 12] It is a diagram showing a processing mode during the execution of the algorithm of neural network processing according to Embodiment 3. [Figure 13] It is a diagram showing an example of a parameter corresponding to a variation of an algorithm for performing a sum-of-products operation process according to Embodiment 4. [Figure 14] It is a diagram showing an example of a parameter corresponding to a variation of an algorithm for performing an activation function process according to Embodiment 4. [Figure 15] It is a diagram showing an example of a combination of parameters of a sum-of-products operation process and an activation function process for each layer of neural network processing according to Embodiment 4. [Figure 16] It is a diagram showing the result of selecting a combination of parameters excluding duplicates from the combinations of parameters according to Embodiment 4. [Figure 17] It is a diagram showing an example of the hardware of a computer that executes software creation processing according to Embodiment 4. [Figure 18] It is a flowchart showing software creation processing according to Embodiment 4.

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings. The following embodiments are examples for explaining the present invention including the drawings, and for the sake of clarity of explanation, omissions and simplifications are made as appropriate. The present invention can be implemented in various other forms. Unless otherwise limited, each component may be singular or plural.

[0012] In the following embodiments, the processes executed by a program may be described. Computers such as servers and clients execute a program by a processor (e.g., a CPU (Central Processing Unit), a GPU), and perform the processing steps defined in the program while using storage resources (e.g., memory) and interface devices (e.g., communication ports), etc. Therefore, the entity that performs the processing by executing the program may be the processor. Similarly, the entity that performs the processing by executing the program may be a controller, a device, a system, a computer, or a node having a processor. The entity that performs the processing by executing the program may be an arithmetic unit and may include a dedicated circuit for performing a specific process. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a CPLD (Complex Programmable Logic Device), a quantum computer, or the like.

[0013] The program may be installed in a computer from a program source. The program source may be, for example, a program distribution server or a storage medium readable by a computer. When the program source is a program distribution server, the program distribution server includes a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Also, in the embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0014] In the following embodiments, the "YYY (processing) unit", "YYY step", and "YYY procedure" are corresponding components of different categories of a device (or system), a method, and a program. For example, a processor or a computer that executes the "YYY step" corresponds to the "YYY processing unit". Also, the "YYY procedure" executed by the processor or the computer corresponds to the "YYY step".

[0015] In the following embodiments, if there are multiple components having the same or similar function, they may be described using the same reference numeral with different subscripts. Furthermore, if it is not necessary to distinguish between these multiple components, the subscripts may be omitted.

[0016] In the following embodiments, "software" may be a component including other files such as programs and libraries, or it may be the program itself.

[0017] [Embodiment 1] (Configuration of the information processing device 10 according to Embodiment 1) Figure 1 shows the configuration of an information processing device 10 according to Embodiment 1. The information processing device 10 is composed of a microprocessor 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, 15, and a GPU 14.

[0018] The microprocessor 11 comprises a CPU 11a and interface circuits 11b1, 11b2, 11b3, 11b4, and 11b5.

[0019] ROM 12 is connected to the microprocessor 11 via interface circuit 11b1. RAM 13 is connected to the microprocessor 11 via interface circuit 11b2. GPU 14 is connected to RAM 15. In addition, GPU 14 is connected to the microprocessor 11 via interface circuit 11b3.

[0020] Cameras 111a and 111b are connected to the microprocessor 11 via interface circuit 11b4. Display device 112 is connected to the microprocessor 11 via interface circuit 11b5.

[0021] The information processing device 10 receives image data from cameras 111a and 111b, performs recognition using neural network processing, and then displays the recognition result on the display device 112. The recognition result may also be output as sound from a speaker. In this embodiment, an example of image recognition is shown, but it is not limited to image recognition and can be applied to other neural network processing in general, such as speech recognition.

[0022] Cameras 111a and 111b each capture images in different directions. This makes it possible to capture images of the entire area to be recognized and perform recognition processing. Alternatively, cameras with a high pixel count and wide field of view may be used as cameras 111a and 111b. This makes it possible to cover the entire area with a small number of cameras, even when the area to be recognized is large. In this case, the image captured by one camera may be divided into multiple areas and recognition processing may be performed separately for each area. This makes it possible to exclude areas that do not need to be recognized, such as the sky, and perform recognition processing only on the necessary areas.

[0023] The microprocessor 11 is a Large Scale Integration (LSI) that integrates the CPU 11a and interface circuits 11b1, 11b2, 11b3, 11b4, and 11b5 onto a single chip. This configuration is just one example, and some or all of the ROM 12, RAM 13, GPU 14, and RAM 15 may also be built into the microprocessor 11.

[0024] The CPU 11a reads and executes the software stored in the ROM 12 via the interface circuit 11b1. Since the read / write speed of the ROM 12 is often slower than that of the RAM 13, the software may be copied from the ROM 12 to the RAM 13 at startup, and then the software is read from the RAM 13 thereafter. The CPU 11a performs the following series of processes according to the software read from either the ROM 12 or the RAM 13.

[0025] First, the CPU 11a acquires image data from cameras 111a and 111b via interface circuit 11b4 and stores it in RAM 13 via interface circuit 11b2. Next, the CPU 11a reads the image data stored in RAM 13 via interface circuit 11b2 and transfers it to the GPU 14 via interface circuit 11b3.

[0026] Next, the CPU 11a reads the software stored in the ROM 12 via the interface circuit 11b1, or in the RAM 13 via the interface circuit 11b2, and transfers it to the GPU 14 via the interface circuit 11b3. Then the CPU 11a instructs the GPU 14 to start the software-based calculation.

[0027] Next, when the CPU 11a receives notification from the GPU 14 that the calculation has finished via the interface circuit 11b3, it retrieves the calculation result from the GPU 14 via the interface circuit 11b3 and stores it in the RAM 13 via the interface circuit 11b2.

[0028] Next, the CPU 11a reads the calculation result from the GPU 14 from the RAM 13 via the interface circuit 11b2, performs predetermined processing, and then displays it on the display device 112 via the interface circuit 11b5.

[0029] When the GPU 14 receives image data from the CPU 11a via the interface circuit 11b3, it stores it in the RAM 15. Also, when the GPU 14 receives software from the CPU 11a via the interface circuit 11b3, it executes the software and stores the calculation results in the RAM 15.

[0030] Furthermore, when the GPU 14 receives a request from the CPU 11a to read the calculation result via the interface circuit 11b3, it reads the calculation result from the RAM 15 and outputs it to the CPU 11a via the interface circuit 11b3.

[0031] (Neural network processing) Figure 2 shows the configuration of a neural network processing. Note that Figure 2 describes the recognition process for a single input image; when performing recognition processing on multiple images, the process shown in Figure 2 should be executed for each input image. In this case, the processing configuration may be the same for all images, or it may differ for each image according to the purpose of the recognition processing.

[0032] The neural network processing shown in Figure 2 consists of N layers (Layer 1, Layer 2, ..., Layer N). Layer 1 takes the input image as input and outputs the processing result to Layer 2. Layer (i+1) (i=1,2,...,N-2) receives data from the previous layer i and outputs the processing result to Layer (i+2). Layer N receives data from the previous layer (N-1) and outputs the processing result as the recognition result of the input image.

[0033] Each of the N layers i consists of a convolution operation and an activation function operation. Details of the convolution operation and activation function operation will be described later.

[0034] (Layer processing of neural network processing related to the first prior art, which is a comparative example) Here, we will describe a neural network processing method according to the first prior art, which is a comparative example of Embodiment 1. Figure 3 is a diagram showing an algorithm for processing one layer of the neural network processing method according to the first prior art. The hardware that performs the neural network processing method according to the first prior art is the same as the information processing device 10 of Embodiment 1.

[0035] When CPU 11a executes the layer processing (layer_n()) for the nth layer (n=1,2,···,N) shown in Figure 3, it activates the convolution process (convolution_n()) in lines 7-20 and the activation function process (activation_n()) in lines 22-32. In the convolution process (convolution_n()), a sum-of-products operation is performed between the input data (in[]) and the weight data (k[]) to generate intermediate data (d[]). In the activation function process (activation_n()), the intermediate data (d[]) is used as input to calculate the value of a predetermined function (f()) and generate output data (out[]).

[0036] The functions `convolution_n()` and `activation_n()` are executed on GPU14. However, they may also be executed on CPU11a.

[0037] The argument "H" in convolution_n() and activation_n() indicates the number of pixels in the vertical direction of the image. The loop processing for the variable h in lines 9-19 of Figure 3 shows the processing for each vertical coordinate of the image. The argument "W" indicates the number of pixels in the horizontal direction of the image. The loop processing for the variable "w" in lines 10-18 of Figure 3 shows the processing for each horizontal coordinate of the image.

[0038] Furthermore, the argument "CO" indicates the number of attribute information types for each pixel in the output data. The loop processing for the variable "co" in lines 11-17 of Figure 3 shows the processing for each type of attribute information for a single pixel in the output data. The value of the argument "CO" is often in the tens to hundreds.

[0039] The argument "CI" indicates the number of attribute information types for each pixel in the input data. The loop processing for the variable "ci" in lines 13-16 of Figure 3 shows the processing for each type of attribute information for a single pixel in the input data. The value of the argument "CI" is typically 3 (corresponding to the three primary colors R (red), G (green), and B (blue)) for Layer 1, which uses the input image as input data. For layers other than Layer 1, the output of the previous layer is used as input, so it is often in the tens to hundreds.

[0040] Furthermore, the argument "k" is the weight data of the layer n being processed.

[0041] Figure 4 shows the execution process of the neural network processing algorithm related to the first prior art. As shown in Figure 4, in the convolution process (convolution_n()), input data (in[]) and weight data (k[]) are read from memory, and intermediate data d[] is written to memory.

[0042] The memory includes a cache memory built into the processing unit (GPU14 in this embodiment) and external memory connected to the outside of the processing unit (RAM15 in this embodiment). For example, when performing image recognition processing, the input data (in[]) and intermediate data (d[]) are large in size and cannot be stored in the cache memory, so access to the external memory becomes necessary.

[0043] The inventors of this application have found that data transfer between the processing unit and external memory takes longer than data transfer to cache memory, and that the processing time for data transfer to and from external memory is a significant amount compared to the arithmetic operation itself. The same applies to activation function processing (activation_n()), where the processing time for data transfer between the processing unit and external memory is a significant amount compared to the arithmetic operation itself.

[0044] In other words, in the first conventional technology, data transfer between the GPU14 and RAM15 occurs during the reading of input data (in[]) for the convolution process (convolution_n()), the writing of intermediate data (d[]), the reading of intermediate data (d[]) for the activation function process (activation_n()), and the writing of output data (out[]), which was found to be a bottleneck in neural network processing performed by software on general-purpose hardware such as the GPU14.

[0045] (Layer processing of neural network processing according to Embodiment 1) Figure 5 shows an algorithm for processing one layer of the neural network processing according to Embodiment 1. When CPU 11a executes the layer processing (layer_n()) shown in Figure 5, it only invokes a process (convact_n()) that combines convolution processing and activation function processing, compared to Figure 3.

[0046] As shown in Figure 5, within the loop processing for the variable "co" in lines 10-16, each time the loop processing for the variable "ci" in lines 12-14 is completed, the activation function processing in line 15 is performed on the intermediate data (d) to generate the output data (out[]).

[0047] Figure 6 shows the execution mode of the neural network processing algorithm according to Embodiment 1. Unlike the first conventional algorithm shown in Figure 3, the neural network processing algorithm according to Embodiment 1 shown in Figure 5 stores intermediate data in a one-dimensional variable d that is not an array. Therefore, it can be stored internally in the GPU 14 (e.g., in cache memory), eliminating the need for data transfer between the GPU 14 and RAM 15. As a result, the amount of data transferred between the GPU 14 and RAM 15 is reduced, and processing time can be shortened.

[0048] [Embodiment 2] (Layer processing of neural network processing related to the second prior art, which is a comparative example) First, we will explain the neural network processing of the second prior art, which is a comparative example of Embodiment 2. Figure 7 shows an algorithm for processing one layer of the neural network processing of the second prior art. In the algorithm of the first prior art in Figure 3, the convolution process consists only of multiply-accumulate operations. On the other hand, in the algorithm of the second prior art in Figure 7, the bias value addition process in lines 28-38 is added to the convolution process in lines 13-38 of Figure 7. The bias value addition process is a process that adds the bias value for each layer of a neural network consisting of N layers to the result of the multiply-accumulate operation. It is known that this can improve the recognition result in some cases.

[0049] Figure 8 shows the processing mode during execution of the neural network processing algorithm related to the second prior art. Compared to Figure 4, the processing of reading the input data (b[], d1[]) for bias addition from RAM15 and writing the result of the bias addition calculation (d2[]) to RAM15 has been added, which increases the processing time for data transfer between GPU14 and RAM15.

[0050] (Layer processing of neural network processing according to Embodiment 2) Figure 9 shows an algorithm for processing one layer of the neural network processing according to Embodiment 2. In this embodiment, the multiply-accumulate operation (multiadd_n() in lines 13-26 of Figure 7), bias addition (Biasadd_n() in lines 28-38 of Figure 7), and activation function processing (activation_n() in lines 40-50 of Figure 7) are executed in a single loop (lines 8-19 of Figure 9).

[0051] As shown in Figure 9, within the loop processing for the variable "co" in lines 10-17, after each completion of the loop processing for the variable "ci" in lines 12-14, the intermediate data (d2) is calculated by adding the bias value (b[]) to the intermediate data (d1), as shown in line 15. Then, as shown in line 16, the activation function processing in line 15 is applied to the intermediate data (d2) to generate the output data (out[]).

[0052] Figure 10 shows the processing mode during execution of the neural network processing algorithm according to Embodiment 2. Unlike the second prior art algorithm shown in Figure 7, the neural network processing algorithm according to Embodiment 2 shown in Figure 9 stores two types of intermediate data in one-dimensional variables d1 and d2, which are not arrays. Therefore, it is possible to hold the intermediate data d1 and d2 inside the GPU 14 (for example, in cache memory), eliminating the need for data transfer between the GPU 14 and RAM 15. Thus, the amount of data transferred between the GPU 14 and RAM 15 is reduced, and processing time can be shortened.

[0053] [Embodiment 3] (Layer processing of neural network processing according to Embodiment 3) Figure 11 shows an algorithm for processing one layer of the neural network processing according to Embodiment 3. In the neural network processing algorithm according to Embodiment 3, as shown in Figure 11, unlike the algorithm of Embodiment 2 in Figure 9, only the sum-of-accumulate operation is executed in a different loop (lines 8-19), while the bias value addition and activation function processing are executed in a single loop (lines 24-32).

[0054] Generally, multiply-accumulate operations are computationally intensive and occupy a large proportion of the overall neural network processing time. Therefore, optimized libraries provided by GPU manufacturers and others are often used to reduce processing time. In the case of the algorithm in Embodiment 2 of Figure 9, bias value addition and activation function processing are incorporated into the same loop processing as the multiply-accumulate operation, so a multiply-accumulate library cannot be used, and the processing time for the multiply-accumulate operation may be longer than when a library is used. On the other hand, in the case of the algorithm in this embodiment of Figure 11, it is possible to replace only the multiply-accumulate processing part with a library, thereby preventing an increase in the processing time for the multiply-accumulate operation.

[0055] Figure 12 shows the execution mode of the neural network processing algorithm according to Embodiment 3. Compared to the second conventional technology in Figure 8, the neural network processing according to this embodiment does not require writing the processing result (d2) of the bias value addition process (biasadd-n()) to and reading it from memory, thus reducing the amount of data transferred between the GPU 14 and RAM 15 and shortening the processing time. Furthermore, compared to Embodiment 2 in Figure 10, the neural network processing according to this embodiment increases the amount of data transferred between the GPU 14 and RAM 15 and the processing time because it involves writing and reading the processing result (d1[]) of the multiply-accumulate operation (multiadd-n()) to and from memory. However, it is possible to use a library for the multiply-accumulate operation (multiadd-n()), which prevents an increase in the processing time of the multiply-accumulate operation, thus potentially reducing the total processing time compared to Embodiment 2 in Figure 10.

[0056] [Embodiment 4] In the embodiments disclosed herein, the convolution process and the activation function process are executed in a single loop. Therefore, if there are multiple variations for both the convolution process and the activation function process, it may be necessary to write a number of processes equivalent to the product of the number of variations when creating the neural network processing software, which could result in an enormous amount of time being spent on software creation and operation verification.

[0057] Regarding convolution operations, it is known that there are many variations in the algorithms for the sum-of-accumulate operation, which is a part of it. Figure 13 shows examples of variations in the algorithm for performing the sum-of-accumulate operation according to Embodiment 4 and the corresponding parameters.

[0058] In Figure 13, “INPUT FORM” indicates the order in which the input data is arranged, and there are two types: “CHW” and “HWC”. In “CHW”, the data is first arranged along the attribute information type axis, then data with the same value along the attribute information type axis are arranged along the image vertical axis, and finally data with the same value along both the attribute information type axis and the image vertical axis are arranged along the image horizontal axis.

[0059] "HWC" first arranges data along the vertical axis of the image, then arranges data along the horizontal axis of the image where the values ​​on the vertical axis are the same, and finally arranges data along the attribute information type axis where the values ​​on the vertical axis and the horizontal axis are the same.

[0060] Note that the algorithms shown in Figures 3, 5, 7, 9, and 11 are software compatible with the input format "HWC" and cannot be applied to the input format "CHW".

[0061] In Figure 13, "WEIGHT FORM" indicates the order in which the weight data is arranged, and there are two types: "CHW" and "HWC". The meanings of "CHW" and "HWC" are the same as those for the input format. The algorithms shown in Figures 3, 5, 7, 9, and 11 are software that supports the "HWC" weight format and cannot be applied when the weight format is "CHW".

[0062] In Figure 13, "OUTPUT FORM" indicates the order in which the output data is sorted, and there are two options: "CHW" and "HWC". The meanings of "CHW" and "HWC" are the same as those for the input format. The algorithms shown in Figures 3, 5, 7, 9, and 11 are software that corresponds to the "HWC" output format and cannot be applied when the output format is "CHW".

[0063] In Figure 13, "WEIGHT SIZE" indicates the range of input data pixels that influence each pixel of the output data. In the algorithms shown in Figures 3, 5, 7, 9, and 11, each pixel of the output data is influenced only by pixels in the input data whose vertical and horizontal axis values ​​are the same. This is denoted as "1x1" in Figure 13. In addition to this, there is a calculation method in which each pixel is influenced by input data in a 3x3 area, which extends the range by one pixel above and below the pixels with the same vertical and horizontal axis values. This is denoted as "3x3" in Figure 13.

[0064] In Figure 13, "CI" represents the number of attribute information types for each pixel in the input data. As explained in Figure 3, it can be 3 for Layer 1 and several tens to several hundred for the others. Due to space limitations, only "3" and "32" are shown here.

[0065] In Figure 13, "CO" represents the number of attribute information types for each pixel in the output data, and as explained in Figure 3, it can take values ​​ranging from tens to hundreds. Here, for space reasons, only one value, "32," is shown.

[0066] Figure 13 does not necessarily cover all variations of algorithms for performing multiply-accumulate operations, but there are still 32 variations.

[0067] Figure 14 shows examples of variations of the activation function processing algorithm according to Embodiment 4 and the corresponding parameters. Figure 14 lists the names of each activation function, but omits the calculation formulas for the functions. While Figure 14 may not cover all variations, there are still 10 variations.

[0068] Multiplying the variations of the multiply-accumulate operation shown in Figure 13 by the variations of the activation function operation shown in Figure 14 results in 320 combinations. This means that in order to execute the convolution operation and the activation function operation in a single loop, it is necessary to write 320 different operations. Therefore, it is estimated that creating and verifying the operation of neural network processing software that includes the multiply-accumulate operation variations shown in Figure 13 and the activation function operation variations shown in Figure 14 will take an enormous amount of time.

[0069] To address this challenge, one approach is to write code that only includes the necessary combinations of code, depending on the application. The development flow for this approach is described below.

[0070] Figure 15 shows an example of parameter sets for the sum-of-accumulate operation and activation function processing for each layer of the neural network processing according to Embodiment 4. While complex recognition processing often involves more than 100 layers, this example shows 30 layers due to space limitations. As can be seen from Figure 15, multiple layers have identical parameter combinations.

[0071] Figure 16 shows the result of selecting parameter combinations that have been eliminated from the parameter combinations related to Embodiment 4. As shown in Figure 16, it can be confirmed that the number of parameter combinations can be narrowed down to three types by selecting parameter combinations that have been eliminated from the parameter combinations in Figure 15. Therefore, it can be seen that if the convolution process and the activation function process are executed in a single loop process, it is sufficient to write three different software programs.

[0072] When a GPU manufacturer provides a library, it is necessary to create software descriptions that support all 320 variations, as it is unknown which combination of convolution and activation function the user will use. Such a vast number of variations can become a bottleneck, potentially making library creation difficult. On the other hand, when a user implements the embodiments disclosed herein, they only need to describe the processing for the combinations necessary depending on the application. For example, as shown in Figure 16, they only need to create software by narrowing it down to three combinations of parameters, which significantly reduces the time required to create the software description.

[0073] (Hardware of the computer 100 that executes the software creation method according to Embodiment 4) Figure 17 is a hardware diagram showing an example configuration of a computer 100 that executes the software creation method according to Embodiment 4.

[0074] Computer 100 is a computer comprising a CPU and other processors 101, a main memory 102, an auxiliary memory 103, a network interface 104, an input device 105, and an output device 106, all interconnected via internal communication lines 109 such as a bus.

[0075] The processor 101 controls the operation of the entire computer 100. The main memory 102 is composed of, for example, volatile semiconductor memory and is used as the work memory of the processor 101. The auxiliary storage device 103 is composed of a large-capacity non-volatile storage device such as a hard disk drive, SSD (Solid State Drive), or flash memory and is used to retain various programs and data for long periods of time.

[0076] A software creation device is realized in which a software creation method is executed by loading a software creation program 103a stored in an auxiliary storage device 103 into the main storage device 102 when the computer 100 starts up or when necessary, and the processor 101 executes the software creation program 103a loaded into the main storage device 102.

[0077] The software creation program 103a may be recorded on a non-temporary recording medium, read from the non-temporary recording medium by a media reader, and loaded into the main memory 102. Alternatively, the software creation program 103a may be obtained from an external computer via a network and loaded into the main memory 102.

[0078] The network interface 104 is an interface device for connecting computer 100 to various networks within the system or for communicating with other computers. The network interface 104 consists of, for example, a NIC (Network Interface Card) such as a wired LAN (Local Area Network) or a wireless LAN.

[0079] The input device 105 consists of a keyboard, a pointing device such as a mouse, and is used by the user to input various instructions and information into the computer 100. The output device 106 consists of a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, or an audio output device such as a speaker, and is used to present necessary information to the user when needed.

[0080] (Software creation process according to Embodiment 4) Figure 18 is a flowchart showing the software creation process according to Embodiment 4.

[0081] First, in step S11, the computer 100 performs a first step of creating a list of combinations of convolution and activation function parameters for each of the multiple layers of processing of the target neural network.

[0082] Next, in step S12, the computer 100 performs a second step of selecting a combination of parameters from the list of parameters created in step S11, excluding duplicates.

[0083] Next, in step S13, the computer 100 determines whether the software to be created is for inference or not. If the software to be created is for inference processing (step S13 YES), the computer 100 moves to step S14, and if it is for learning processing (step S13 NO), it moves to step S15.

[0084] In step S14, the computer 100 performs a third step in which it creates a program that performs the activation function operation within the convolution loop operation, with respect to the convolution operation and activation function operation corresponding to the combination of parameters selected in step S13.

[0085] Meanwhile, in step S15, the computer 100 performs a fourth step in which, with respect to the convolution and activation function operations corresponding to the combination of parameters selected in step S13, it creates a program that executes the activation function operation within a different loop than the convolution operation.

[0086] Furthermore, some or all of the above steps S11 to S15 may be performed manually.

[0087] According to this embodiment, when creating neural network processing software, it is sufficient to narrow down the combination of parameters, thereby significantly reducing the time required for software creation and operational testing.

[0088] (Other embodiments) Furthermore, another approach to shortening the time required to write software is to focus on applications where neural network processing is used. Generally, neural network processing is used for two types of applications: training and inference. In training, known data is input, and weight data is adjusted so that the output data approaches the expected value. On the other hand, inference is input, unknown data is input, and the output data is used as the recognition result. Training is usually performed on servers with abundant computing resources, and the constraints on processing time are lenient. Therefore, the need to shorten processing time is relatively small.

[0089] On the other hand, inference processing is sometimes performed in devices installed in automobiles, for example, where computing resources are limited. Furthermore, when recognition results are used to control devices, such as in autonomous driving, there are relatively strict constraints on processing time in order to control the devices in a timely manner. Therefore, there is a relatively high need to reduce processing time. Accordingly, by applying the embodiments disclosed in this application to the software used for inference processing to reduce processing time, and by using existing software without applying the embodiments disclosed in this application to the software used for learning processing, it is possible to reduce the time required to create the software while meeting the needs.

[0090] In the embodiments disclosed herein, while the creation of software that performs activation processing within the inference convolution loop may be a burden on the user, it achieves the effect of balancing processing speed and recognition accuracy at edge terminals such as in-vehicle devices in automobiles. In the embodiments disclosed herein, the burden on the user of software creation can be reduced by providing a program creation method that narrows down the combination of overlapping parameters in Embodiment 4 and reduces the number of algorithms to be created in the software.

[0091] The present invention is not limited to the embodiments described above and includes various modifications. For example, the embodiments described above are described in detail to illustrate the present invention clearly and are not necessarily limited to those having all the configurations described. Furthermore, as long as they do not contradict each other, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and to add configurations of other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, replace, integrate, or distribute parts of the configuration of each embodiment. Moreover, the configurations and processes shown in the embodiments can be distributed, integrated, or rearranged as appropriate based on processing efficiency or implementation efficiency. [Explanation of Symbols]

[0092] 10: Information processing equipment, 100: Computer

Claims

1. An information processing device having a processor and a memory, The processor, A convolution process for generating a plurality of intermediate data by performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data; Activation function processing to calculate a value of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; Each time the intermediate data is generated in the loop processing of the convolution processing, software is executed to execute the activation function processing using the intermediate data as an input.

23. An information processing apparatus comprising:

2. 2. The information processing device according to claim 1, The processor, In the loop process, each time the multiply-and-accumulate operation is performed to generate one piece of intermediate data among the plurality of intermediate data, the activation function process is performed to calculate a value of the predetermined function using the one piece of intermediate data as an input, thereby generating one piece of output data among the plurality of output data.

23. An information processing apparatus comprising:

3. 2. The information processing device according to claim 1, the convolution process includes a bias value addition process of adding a bias value to a result of the multiply-and-accumulate operation; The bias value addition process is executed in the same loop as the activation function process.

23. An information processing apparatus comprising:

4. An information processing device having a processor and a memory, The processor, A convolution process for generating a plurality of intermediate data by performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data; Activation function processing to calculate a value of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; the convolution process includes a bias value addition process of adding a bias value to a result of the multiply-and-accumulate operation; The processor, The bias value addition process is executed in the same loop process as the activation function process; The bias value addition process and the activation function process are executed in a loop process different from the convolution process.

23. An information processing apparatus comprising:

5. The information processing device according to any one of claims 1 to 4, the plurality of input data are a plurality of image data corresponding to a plurality of ranges, The output data is a result of image recognition processing on the plurality of image data.

23. An information processing apparatus comprising:

6. An information processing method executed by an information processing device having a processor and a memory, a convolution processing step in which the processor performs a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing step in which the processor calculates values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; A neural network processing step including: The processor executes software written to execute the activation function processing step using the intermediate data as input each time the intermediate data is generated in the loop processing of the convolution processing step.

23. An information processing method comprising:

7. An information processing method executed by an information processing device having a processor and a memory, a convolution processing step in which the processor performs a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing step in which the processor calculates values ​​of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; A neural network processing step including: the convolution processing step includes a bias value addition processing step in which the processor adds a bias value to a result of the multiply-accumulate operation; The processor, The bias value addition step is executed in the same loop processing as the activation function processing step; The bias value addition step and the activation function processing step are executed in a loop process different from the convolution processing step.

23. An information processing method comprising:

8. A computer having a processor and a memory, A convolution processing procedure for performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing step for calculating a value of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; Executing a neural network processing procedure including: Each time the intermediate data is generated in the loop processing of the convolution processing procedure, software is executed that is written to execute the activation function processing procedure using the intermediate data as input.

2. An information processing program comprising:

9. A computer having a processor and a memory, A convolution processing procedure for performing a multiply-and-accumulate operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data; an activation function processing step for calculating a value of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data; Executing a neural network processing procedure including: the convolution process step includes a bias value addition process step of adding a bias value to a result of the multiply-and-accumulate operation; The processor, Executing the bias value addition process in the same loop as the activation function process; The bias value addition process and the activation function process are executed in a loop different from the convolution process.

2. An information processing program comprising:

10. A software creation device that creates software for causing an information processing device to execute neural network processing for performing recognition processing of input data by combining a plurality of layer processes each including a convolution process that performs a product-sum operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data, and an activation function process that uses the plurality of intermediate data as input and calculates a value of a predetermined function to generate a plurality of output data, the software creation device comprising: A first processing unit that creates a list of combinations of parameters of the convolution processing and the activation function processing for each of the layer processing; A second processing unit that selects a combination of parameters from the list, excluding duplicates; a third processing unit that creates software for executing the activation function processing within a loop process of the convolution processing, with respect to the convolution processing and the activation function processing corresponding to the combination of parameters selected by the second processing unit; A software creation device comprising:

11. 11. A software production apparatus according to claim 10, The third processing unit is When creating software for an inference process in which unknown data is input as the plurality of input data and the plurality of output data is used as a recognition result, software is created for the convolution process and the activation function process corresponding to the combination of parameters selected by the second processing unit, the activation function process being executed within a loop process of the convolution process; When creating software for a learning process in which known data is input as the plurality of input data and the plurality of weight data is adjusted so that the plurality of output data approaches expected values, software is created for the convolution process and the activation function process corresponding to the combination of parameters selected by the second processing unit, the activation function process being executed in a loop process different from the convolution process. A software creation device comprising:

12. A software creation method for creating software in which a computer having a processor and a memory creates software that causes an information processing device to execute neural network processing for performing recognition processing of the input data by combining a plurality of layer processes each including a convolution process for performing a product-sum operation between a plurality of input data and a plurality of weight data to generate a plurality of intermediate data, and an activation function process for calculating a value of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data, the method comprising the steps of: A first step in which the computer creates a list of combinations of parameters of the convolution process and the activation function process for each of the layer processes; a second step in which the computer selects a combination of parameters from the list with duplicates removed; a third step of the computer creating software for executing the activation function processing within a loop process of the convolution processing, for the convolution processing and the activation function processing corresponding to the combination of the parameters selected by the second step; 13. A method for creating software comprising the steps of:

13. A computer having a processor and a memory, A software creation program for creating software that causes an information processing device to execute neural network processing for performing recognition processing of the input data by combining a plurality of layer processes each including a convolution process for generating a plurality of intermediate data by performing a product-sum operation between a plurality of input data and a plurality of weight data, and an activation function process for calculating a value of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data, the software creation program comprising: The computer includes: A first step of creating a list of combinations of parameters of the convolution process and the activation function process for each of the layer processes; a second step of selecting a combination of parameters from the list, with duplicates removed; a third step of creating software for executing the activation function processing within a loop process of the convolution processing, for the convolution processing and the activation function processing corresponding to the combination of the parameters selected by the second step; A software creation program characterized by executing the above.

14. A software creation method for creating software that causes an information processing device to execute neural network processing for performing recognition processing of input data by combining a plurality of layer processes each including a convolution process for generating a plurality of intermediate data by performing a product-sum operation between a plurality of input data and a plurality of weight data, and an activation function process for calculating a value of a predetermined function using the plurality of intermediate data as input to generate a plurality of output data, the method comprising the steps of: A first step of creating a list of combinations of parameters of the convolution process and the activation function process for each of the layer processes; a second step of selecting a combination of parameters from said list with duplicates removed; a third step of creating software for executing the activation function processing within a loop process of the convolution processing, for the convolution processing and the activation function processing corresponding to the combination of the parameters selected by the second step; 13. A method for creating software comprising the steps of:

Images