Information processing device, vehicle-mounted control device

Abstract

The information processing apparatus of the present invention is an information processing apparatus that performs DNN operations under a neural network composed of multiple layers. For each of a first region and a second region different from the first region in a feature map input to the neural network, it performs arithmetic processing corresponding to a specified layer of the neural network, and integrates the result of the arithmetic processing for the first region and the result of the arithmetic processing for the second region, and outputs the integrated result as the result of the arithmetic processing for the feature map.

Description

Technical Field

[0001] The present invention relates to an information processing device and a vehicle control device using the same. Background Art

[0002] Conventionally, a technique has been widely used in which the surrounding conditions of a vehicle are recognized using a photographed image of a camera and information from various sensors, and various driving assistances are performed based on the recognition result. In such a vehicle driving assistance technique, in recent years, in order to obtain a highly accurate recognition result for complex surrounding conditions, a method of performing an operation using a neural network, which is a functional model of nerve cells in the human brain, has been proposed.

[0003] Generally, in an information processing device (ECU: Electronic Control Unit) to be mounted in a vehicle, in order to perform an operation using a neural network, due to the constraint that the ECU is driven using power supplied from an in-vehicle battery, low power consumption is required. Therefore, an arithmetic circuit having a relatively small internal memory capacity, such as a small-scale FPGA (Field Programmable Gate Array), is mostly used.

[0004] In an arithmetic circuit with a small internal memory capacity, there is a case where intermediate data generated during the operation cannot be stored in the internal memory. In such a case, at least a part of the intermediate data must be stored in an external storage device provided outside the arithmetic circuit, and then read from the external storage device when the arithmetic circuit needs it. However, the data transfer speed between the arithmetic circuit and the external storage device is usually slower than the data transfer speed of the internal memory. Therefore, a problem of reduced processing speed occurs.

[0005] As a technique for solving the above problems, Patent Document 1 is known. Patent Document 1 discloses a convolutional calculation method in a neural network, including the following steps: performing depthwise convolution calculation and pointwise convolution calculation based on an input feature map read from a DRAM, a depthwise convolution kernel, and a pointwise convolution kernel, and obtaining output feature values of a first specified number p of points on all pointwise convolution output channels; and repeating the above operation to obtain output feature values of all points on all pointwise convolution output channels. And it is described that: thereby, the storage area for storing intermediate results can be reduced.

[0006] Prior Art Documents

[0007] Patent Documents

[0008] Patent Document 1: Japanese Patent Application Publication No. 2019-109895 Summary of the invention

[0009] Problem that the invention aims to solve

[0010] In the technology of Patent Document 1, the convolution calculation in the neural network is divided into two convolution calculations, namely, depthwise convolution calculation and pointwise convolution calculation. Therefore, there is a problem that part of the information is lost when the intermediate results are handed over between these convolution calculations, thereby causing the degradation of recognition accuracy.

[0011] Technical means of solving problems

[0012] An information processing device in one form of the present invention is an information processing device that performs DNN operations under a neural network composed of multiple layers, which performs operation processing corresponding to a specified layer of the neural network for each of a first region in a feature map input to the neural network and a second region different from the first region, and integrates the result of the operation processing on the first region with the result of the operation processing on the second region and outputs it as the result of the operation processing on the feature map.

[0013] Another form of an information processing device of the present invention is an information processing device that performs DNN operations under a neural network composed of multiple layers, and comprises: a feature map segmentation unit that segments a feature map input to the neural network into multiple regions in such a way that each of the segmented regions contains a mutually repeated redundant portion; an NN operation unit that is provided corresponding to each layer of the neural network and performs a prescribed operation process on each of the multiple regions; an internal storage unit that stores the results of the operation process performed by the NN operation unit; and a feature map integration unit that integrates the results of the operation process performed by the NN operation unit corresponding to the prescribed layer of the neural network on the multiple regions. The rows are integrated and stored in an external storage device provided outside the information processing device, the size of the redundant part is determined according to the size and stride of the filter used in the operation processing, the number of divisions of the feature map by the feature map segmentation unit and the number of layers of the neural network before the NN operation unit performs the operation processing before integrating the results of the operation processing by the feature map integration unit are determined according to at least one of the storage capacity of the internal storage unit, the total amount of operation processing performed by the NN operation unit, the data transmission frequency band between the information processing device and the external storage device, and the change in the data size before and after the operation processing performed by the NN operation unit.

[0014] The vehicle control device of the present invention includes the above information processing device and an action plan formulation unit that formulates an action plan for the vehicle. The information processing device performs the arithmetic processing based on sensor information related to the surrounding conditions of the vehicle, and the action plan formulation unit formulates the action plan for the vehicle based on the result of the arithmetic processing output from the information processing device.

[0015] Effects of the Invention

[0016] According to the present invention, in an information processing device that performs arithmetic operations using a neural network, it is possible to achieve high-speed processing without deterioration of recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A diagram showing the configuration of a vehicle control device according to an embodiment of the present invention.

[0018] Figure 2 A diagram showing the configuration of a DNN arithmetic device according to an embodiment of the present invention.

[0019] Figure 3 A functional block diagram of each NN arithmetic unit of the arithmetic processing unit according to an embodiment of the present invention.

[0020] Figure 4 A diagram showing an outline of the arithmetic processing performed by a DNN arithmetic device according to an embodiment of the present invention.

[0021] Figure 5 A diagram for explaining a method of setting a redundant part in a feature map segmentation unit.

[0022] Figure 6 A flowchart showing an example of a process for determining the number of divisions of a feature map and the storage location of intermediate data. DETAILED DESCRIPTION OF THE INVENTION

[0023] Figure 1 A diagram showing the configuration of a vehicle control device according to an embodiment of the present invention. Figure 1 The vehicle control device 1 shown is used by being mounted on a vehicle and is connected to a camera 2, a LiDAR (Light Detection and Ranging) 3, and a radar 4 that respectively function as sensors for detecting the surrounding conditions of the vehicle. The photographed image of the vehicle surroundings obtained by the camera 2 and the distance information from the vehicle to surrounding objects obtained by the LiDAR 3 and the radar 4 are input to the vehicle control device 1. Furthermore, a plurality of cameras 2, LiDARs 3, and radars 4 may be mounted on the vehicle, and the photographed images and distance information respectively obtained by these plurality of sensors are input to the vehicle control device 1.

[0024] The in-vehicle control device 1 includes functional blocks such as a DNN arithmetic device 10, a sensor fusion unit 11, a feature map storage unit 12, an external storage device 13, and an action plan creation unit 15. The DNN arithmetic device 10, the sensor fusion unit 11, and the action plan creation unit 15 are respectively constituted by using arithmetic processing circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc., and various programs used in combination with them. In addition, the feature map storage unit 12 and the external storage device 13 are respectively constituted by using storage devices such as a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a flash memory. Furthermore, the DNN arithmetic device 10 performs information processing for recognizing the surrounding situation of the vehicle by executing DNN arithmetic under a neural network composed of multiple layers, and is equivalent to the information processing device of an embodiment of the present invention.

[0025] The captured images and distance information respectively input from the camera 2, the LiDAR 3, and the radar 4 are stored in the feature map storage unit 12 in the form of a feature map that uses the respective pixel values on a two-dimensional plane to represent the features related to the surrounding situation of the vehicle. Furthermore, the distance information respectively input from the LiDAR 3 and the radar 4 is converted into a feature map through sensor fusion processing by means of the sensor fusion unit 11 and stored in the feature map storage unit 12. However, it is not necessary to perform sensor fusion processing. In addition, a feature map based on information from other sensors may be further stored in the feature map storage unit 12, or only one of the captured image and the distance information may be stored in the feature map storage unit 12 in the form of a feature map.

[0026] The DNN operation device 10 reads out a feature map (photographed image or distance information) from the feature map storage unit 12 and performs DNN (Deep Neural Network) operations on the read feature map. The so-called DNN operations performed by the DNN operation device 10 are an operation process equivalent to a form of artificial intelligence, and the functions of a neural network composed of multiple layers are realized through the operation process. When performing DNN operations, the DNN operation device 10 acquires the required weight information from the external storage device 13. The external storage device 13 stores the weight information that has been pre-calculated by a server (not shown) and updated according to the learning results of the DNN operations performed by the DNN operation device 10 up to now as a learned model. Furthermore, the details of the DNN operation device 10 will be described later.

[0027] The action plan formulation unit 15 formulates an action plan for the vehicle based on the DNN operation result given by the DNN operation device 10 and outputs action plan information. For example, information for assisting the braking operation and steering wheel operation performed by the driver of the vehicle, and information for the vehicle to perform autonomous driving are output as the action plan information. The content of the action plan information output from the action plan formulation unit 15 is displayed on a display provided inside the vehicle, or input to various ECUs (Electronic Control Units) installed in the vehicle for various vehicle controls. Furthermore, the action plan information can also be sent to a server or other vehicles.

[0028] Next, the DNN operation device 10 will be described. Figure 2 FIG. showing the configuration of the DNN operation device 10 according to an embodiment of the present invention. As Figure 2 shown, the DNN operation device 10 includes a feature map segmentation unit 101, an operation processing unit 102, a feature map integration unit 103, and an internal storage unit 104.

[0029] The feature map segmentation unit 101 divides the feature map read from the feature map storage unit 12 and input to the DNN operation device 10 into a plurality of regions. Furthermore, the details of the method for dividing the feature map by the feature map segmentation unit 101 will be described later.

[0030] The operation processing unit 102 sequentially performs the above-described DNN operations on each region segmented from the feature map by the feature map segmentation unit 101. In the operation processing unit 102, N NN operation units are arranged in a layered manner, from the first-layer NN operation unit 102-1 to the Nth-layer NN operation unit 102-N (where N is a natural number of 3 or more). That is, in the operation processing unit 102, an N-layer neural network is formed by the first-layer NN operation unit 102-1, the second-layer NN operation unit 102-2, ···, the kth-layer NN operation unit 102-k, ···, the Nth-layer NN operation unit 102-N. The operation processing unit 102 performs DNN operations by respectively setting weights for these NN operation units provided corresponding to each layer of the neural network, thereby calculating an operation result representing the recognition result of the surrounding situation of the vehicle using each region of the feature map. Furthermore, Figure 2 Among the N-layer NN operation units shown, the first first-layer NN operation unit 102-1 corresponds to the input layer, and the last Nth-layer NN operation unit 102-N corresponds to the output layer.

[0031] The operation results given by the NN operation units of each layer in the operation processing unit 102 are stored as intermediate data in the internal storage unit 104 or the external storage device 13 and handed over to the next-layer NN operation unit. That is, the NN operation units of each layer except the input layer read the intermediate data representing the operation results given by the previous-layer NN operation unit from the internal storage unit 104 or the external storage device 13, and use this operation result to perform operation processing corresponding to the specified layer of the neural network.

[0032] The feature map integration unit 103 integrates the operation results of each region obtained by sequentially performing DNN operations on each region by the operation processing unit 102, outputs them as the operation result of the DNN operation device 10, and stores them in the external storage device 13. Thus, the DNN operation result for the feature map input to the DNN operation device 10 is obtained and can be used in the action plan formulation unit 15 for formulating the action plan of the vehicle.

[0033] Figure 3 It is a functional block diagram of each NN operation unit of the operation processing unit 102 according to an embodiment of the present invention. Furthermore, in the operation processing unit 102, the first-layer NN operation unit 102-1 to the Nth-layer NN operation unit 102-N all have the same functional configuration, so Figure 3 the functional blocks of the kth-layer NN operation unit 102-k are shown in a representative manner for them. Next, all the NN operation units constituting the operation processing unit 102 of the present embodiment will be described by explaining the functional blocks of the kth-layer NN operation unit 102-k.

[0034] The k-th layer NN operation unit 102-k includes a convolution processing unit 121, an activation processing unit 122, and a pooling processing unit 123.

[0035] The input data from the upper layer (the k-1-th layer) for the k-th layer NN operation unit 102-k is input to the convolution processing unit 121 and the pooling processing unit 123. Furthermore, in the case of the first layer NN operation unit 102-1, each region of the feature map read out from the feature map storage unit 12 and segmented by the feature map segmentation unit 101 is input to the convolution processing unit 121 and the pooling processing unit 123 as the input data from the upper layer.

[0036] The convolution processing unit 121 performs a convolution operation corresponding to the k-th layer of the neural network according to the weight information stored in the external storage device 13 as a learned model. The so-called convolution operation performed in the convolution processing unit 121 is the following operation process: for each position of the filter when a filter (kernel) of a specified size set according to the weight information is moved on the input data at each specified interval, the sum of the products of each pixel of the input data within the filter range and the corresponding filter elements is calculated. Furthermore, the moving interval of the filter at this time is called the stride.

[0037] The activation processing unit 122 performs an activation operation for activating the operation result of the convolution processing unit 121. Here, for example, an activation function called the ReLU (Rectified Linear Unit) function is used to perform the activation operation. The so-called ReLU function is a function that outputs 0 for input values less than 0 and directly outputs the input value for values of 0 or more. Furthermore, an activation operation other than the ReLU function can also be used. Through the activation operation performed by the activation processing unit 122, the data values in the operation result of the convolution processing unit 121 that have little influence on the operation in the next layer (the k+1-th layer) are converted to 0.

[0038] The pooling processing unit 123 performs a pooling operation corresponding to the k-th layer of the neural network. The so-called pooling operation performed in the pooling processing unit 123 is the following operation process: for each position of the filter when a filter of a specified size is moved on the input data at each specified interval, the features of each pixel of the input data within the filter range are extracted. For example, pooling operations such as average pooling that extracts the average value of each pixel within the filter range and max pooling that extracts the maximum value of each pixel within the filter range are known. Furthermore, the moving interval of the filter at this time is also called the stride, the same as that of the convolution processing unit 121.

[0039] Each data value calculated by the convolution operation performed by the convolution processing unit 121 and then subjected to the activation operation by the activation processing unit 122, or each data value calculated by the pooling operation performed by the pooling processing unit 123 is output from the k-th layer NN operation unit 102-k and becomes the input data for the next layer. Here, usually one of the convolution operation or the pooling operation is performed in each layer NN operation unit. In the neural network of the operation processing unit 102, the layer provided with the NN operation unit that performs the convolution operation is also called a "convolution layer", and the layer provided with the NN operation unit that performs the pooling operation is also called a "pooling layer". Furthermore, the pooling processing unit 123 may not be provided in the convolution layer NN operation unit, and the convolution processing unit 121 and the activation processing unit 122 may not be provided in the pooling layer NN operation unit. Or, it may be configured such that each layer NN operation unit has Figure 3 a configuration that enables the convolution layer and the pooling layer to be arbitrarily switched.

[0040] Next, the features of the DNN operation device 10 of the present embodiment will be described. The bandwidth of the data transfer frequency band between the operation processing unit 102 and the external storage device 13 is generally narrower than that of the internal storage unit 104 built in the DNN operation device 10. That is, the data transfer speed between the operation processing unit 102 and the external storage device 13 is slower than that of the internal storage unit 104. Therefore, in order to speed up the DNN operation performed by the DNN operation device 10, it is preferable to store the intermediate data calculated by each layer NN operation unit as much as possible in the internal storage unit 104 rather than in the external storage device 13. However, due to hardware restrictions on the DNN operation device 10, etc., the memory capacity that can be ensured in the form of the internal storage unit 104 is relatively small. Therefore, depending on the data size of the feature map, it may not be possible to store all the intermediate data obtained in each layer NN operation unit in the internal storage unit 104.

[0041] Therefore, in the DNN operation device 10 of the present embodiment, the feature map is divided into a plurality of regions by the feature map division unit 101, and each layer NN operation unit of the operation processing unit 102 sequentially performs operation processing on each of the divided regions. Thus, compared with the case where the feature map is not divided and directly input to the operation processing unit 102, the data size of the intermediate data output from each layer NN operation unit is reduced, so that it can be stored in the internal storage unit 104. Subsequently, the operation results of each region output from the last output layer are integrated in the feature map integration unit 103, thereby obtaining the DNN operation result for the feature map. Thus, even if the memory capacity of the internal storage unit 104 is small, the DNN operation performed by the DNN operation device 10 can be speeded up without deterioration of the recognition accuracy based on the feature map.

[0042] Figure 4A diagram showing an overview of the arithmetic processing performed by the DNN arithmetic unit 10 according to an embodiment of the present invention.

[0043] The feature map 30 input to the DNN arithmetic unit 10 is first divided into a plurality of regions 31 to 34 in the feature map division unit 101. Furthermore, Figure 4 Shown is an example in which each of the three feature maps 30 corresponding to the respective image data of R, G, and B is divided into four, and thus four regions 31 to 34 are generated for each feature map 30. However, the number of feature maps and the number of divisions are not limited thereto. Here, M is an ID for identifying each region, and ID values of M = 1 to M = 4 are sequentially set for the regions 31 to 34.

[0044] The regions 31 to 34 divided from the feature map 30 each contain redundant parts 41 to 44. In the redundant parts 41 to 44, adjacent regions correspond to the same part in the feature map 30 before division. For example, the right part in the redundant part 41 contained in the region 31 and the left part in the redundant part 42 contained in the region 32 correspond to the same part in the feature map 30 before division and are the same content. In addition, the lower part in the redundant part 41 contained in the region 31 and the upper part in the redundant part 43 contained in the region 33 correspond to the same part in the feature map 30 before division and are the same content. That is, the feature map division unit 101 divides the feature map 30 into regions 31 to 34 in such a way that the adjacent regions each contain the redundant parts 41 to 44 that repeat each other.

[0045] Furthermore, the sizes of the redundant parts 41 to 44 set in the feature map division unit 101 are determined according to the sizes and strides of the filters used in the convolution operations and pooling operations respectively performed by the NN arithmetic units 102-1 to 102-N in the arithmetic processing unit 102. This will be described later with reference to Figure 5 for explanation.

[0046] The regions 31 to 34 divided from the feature map 30 by the feature map division unit 101 are input to the arithmetic processing unit 102. In the arithmetic processing unit 102, arithmetic processing using the NN arithmetic units 102-1 to 102-N corresponding to the respective layers of the neural network is sequentially performed on each of the regions 31 to 34. Thus, the DNN arithmetic is executed for each region divided from the feature map 30. That is, when the DNN arithmetic is performed on the region 31 (M = 1) and the output data 51 representing the arithmetic result is obtained, the DNN arithmetic is performed on the next region 32 (M = 2) and the output data 52 representing the arithmetic result is obtained. By sequentially performing such processing on the regions 31 to 34, the output data 51 to 54 corresponding to the DNN arithmetic results can be obtained for each of the regions 31 to 34.

[0047] Furthermore, during the execution of the DNN operation in the arithmetic processing unit 102, the intermediate data obtained in each layer of the NN operation unit is temporarily stored in the internal storage unit 104 and used as the input data for the next layer of the NN operation unit. At this time, the data stored in the internal storage unit 104 is rewritten for each layer of the neural network being processed. In addition, the intermediate data stored in the internal storage unit 104 when the DNN operation is being performed on region 31 is different from the intermediate data stored in the internal storage unit 104 when the DNN operation is being performed on the next region 32. The same applies to regions 33 and 34. That is, the results of the arithmetic processing performed by each layer of the NN operation unit for regions 31 to 34 are stored in the internal storage unit 104 at different times respectively.

[0048] When all the DNN operations in the arithmetic processing unit 102 are completed, the output data 51 to 54 from the output layer for regions 31 to 34 are input to the feature map integration unit 103. In the feature map integration unit 103, the output data 51 to 54 are integrated to generate integrated data 50 representing the DNN operation result for the feature map 30 before segmentation. Specifically, for example, as Figure 4 shown, the output data 51 to 54 based on regions 31 to 34 can be arranged and configured according to the positions when regions 31 to 34 are segmented from the feature map 30 and then synthesized to generate the integrated data 50. The integrated data 50 generated in the feature map integration unit 103 is stored in the external storage device 13.

[0049] Furthermore, the feature map integration unit 103 can not only integrate the operation results of each region output from the output layer of the arithmetic processing unit 102, but also integrate the operation results of each region output from any intermediate layer among the intermediate layers provided between the input layer and the output layer. That is, the feature map integration unit 103 can integrate the results of the arithmetic processing performed by the NN operation unit 102-(k+α) corresponding to the (k+α)-th layer (α is an arbitrary natural number) of the neural network for each region. Furthermore, at this time, the operation results in the intermediate layer stored in the external storage device 13 can also be input to the feature map segmentation unit 101, segmented into multiple regions in the same way as the feature map in the feature map segmentation unit 101, and then input to the next layer of the NN operation unit for arithmetic processing. In this case, the operation results in the intermediate layer integrated by the feature map integration unit 103 are temporarily stored in the external storage device 13 and input from the external storage device 13 to the next layer of the NN operation unit, that is, the NN operation unit 102-(k+α+1) corresponding to the (k+α+1)-th layer of the neural network and used for the arithmetic processing in this layer.

[0050] Next, a method for setting the redundant part in the feature map division unit 101 will be described. In the feature map division unit 101, when the input feature map is divided into multiple regions, the redundant part as described above is set for each region. This redundant part is for the NN operation units 102-1 to 102-N in the operation processing unit 102 to accurately perform their respective convolution operations and pooling operations, that is, to obtain the same result as when performing operations on the feature map before division. Specifically, the redundant part is set as follows according to the size and stride of the filters used in each NN operation unit.

[0051] Figure 5 A diagram for explaining the method of setting the redundant part in the feature map division unit 101. Figure 5 (a) of which shows an example of setting the redundant part when the size of the filter used in the operation processing in the input layer is 3×3, the stride is 1, the size of the filter used in the operation processing in the intermediate layer is 1×1, and the stride is 1. Figure 5 (b) of which shows an example of setting the redundant part when the size of the filter used in the operation processing in the input layer is 3×3, the stride is 1, and the size of the filter used in the operation processing in the intermediate layer is 3×3 and the stride is 2. Furthermore, Figure 5 (a) of which, Figure 5 (b) of which show examples of setting the redundant part in the DNN operation where there is only 1 intermediate layer between the input layer and the output layer for the sake of simplified explanation. In the case of having 2 or more intermediate layers, the redundant part can also be set by the same method.

[0052] In order to accurately perform the operation processing of the input layer on each region after the feature map is divided, it is necessary to obtain the same operation result as before division when applying the filter to the boundary part of each region after division. The same applies to each intermediate layer between the input layer and the output layer. Therefore, in the feature map division unit 101, the size of the redundant part when dividing the feature map into multiple regions is determined so as to satisfy such conditions for the input layer and each intermediate layer.

[0053] In Figure 5 the example of (a), since the size of the filter in the operation processing of the input layer is 3×3 and the stride is 1, a redundant part of 2 pixel amounts needs to be set for the operation processing of the input layer. On the other hand, since the size of the filter in the operation processing of the intermediate layer is 1×1 and the stride is 1, no redundant part needs to be set for the operation processing of the intermediate layer. Thus, it can be known that in the input layer of Figure 5 (a), a redundant part can be set with a width of 2 pixel amounts as shown by the hatching for the boundary part of each region after the feature map is divided. Furthermore, Figure 5The illustration of the vertical redundant part is omitted in (a). The same applies when dividing along the vertical direction. The redundant part can be set with a width of 2 pixels.

[0054] In Figure 5 the example of (b), the size of the filter in the arithmetic processing of the input layer is 3×3 and the stride is 1. Therefore, similar to Figure 5 (a), a redundant part of 2 pixels needs to be set for the arithmetic processing of the input layer. In addition, the size of the filter in the arithmetic processing of the intermediate layer is 3×3 and the stride is 2. Therefore, a redundant part of 1 pixel needs to be set for the arithmetic processing of the intermediate layer. Thus, it can be known that in Figure 5 (b) of the input layer, as shown by the hatching, a redundant part can be set for the boundary part of each region after the feature map is divided with a width of 3 pixels where the input layer and the intermediate layer are combined. Furthermore, Figure 5 the illustration of the vertical redundant part is omitted in (b). The same applies when dividing along the vertical direction. The redundant part can be set with a width of 3 pixels.

[0055] As described above, in the feature map division unit 101, when dividing the feature map input to the arithmetic processing unit 102, the number of pixels of the redundant part required for the arithmetic processing in each layer of the arithmetic processing unit 102 before integrating the output data is accumulated to determine the size of the redundant part for each divided region. Specifically, for example, the width W of the redundant part when dividing the feature map can be determined by the following formula (1). In formula (1), A k represents the filter size of the k-th layer, and S k represents the stride of the k-th layer. In addition, N represents the number of layers of the neural network constituting the arithmetic processing unit 102, that is, the number of NN arithmetic units.

[0056] [Equation 1]

[0057]

[0058] Next, a method for determining the number of divisions of the feature map and the storage location of intermediate data will be described. As described above, the operation results given by each layer NN operation unit in the operation processing unit 102 are stored as intermediate data in the internal storage unit 104 or the external storage device 13. To speed up the DNN operation performed by the DNN operation device 10 of the present embodiment, the memory capacity of the internal storage unit 104 needs to be considered and set in such a way that the intermediate data calculated by each layer NN operation unit constituting the operation processing unit 102 can be stored in the internal storage unit 104 as much as possible. Among them, when the stride number of the filter used in the operation processing of the intermediate layer is 2 or more, the data size after the operation is reduced. Therefore, for the internal storage unit 104, to suppress the required memory capacity, it is preferable to integrate the output data obtained up to the previous layer. The number of divisions of the feature map in the feature map division unit 101 and which of the internal storage unit 104 and the external storage device 13 is used as the storage location of the intermediate data need to be determined considering these conditions.

[0059] Figure 6 FIG. is a flowchart showing an example of the process of determining the number of divisions of the feature map and the storage location of the intermediate data. Furthermore, Figure 6 the process shown in the flowchart can be implemented in the DNN operation device 10, or can be implemented in other parts within the vehicle control device 1. Alternatively, the process shown in the flowchart can also be implemented in advance using a general-purpose computer or the like Figure 6 so as to pre-determine the number of divisions of the feature map and the storage location of the intermediate data in the DNN operation device 10, and determine the specifications of the DNN operation device 10 according to the result.

[0060] In step S10, an initial value k = 1 is set for the NN operation unit 102-k to be processed.

[0061] In step S20, it is determined whether the stride of the NN operation unit 102-(k + 1) of the (k + 1)-th layer, which is the next layer of the NN operation unit 102-k currently selected as the processing object, is 2 or more. When the stride of the (k + 1)-th layer is 2 or more, that is, when the moving interval of the filter used in the operation processing of the NN operation unit 102-(k + 1) is 2 pixels or more, the process proceeds to step S50; otherwise, the process proceeds to step S30.

[0062] In step S30, it is determined whether the size of the output data from the NN operation unit 102-k selected as the current processing target is equal to or less than the memory capacity of the internal storage unit 104. If the size of the output data from the NN operation unit 102-k is equal to or less than the memory capacity of the internal storage unit 104, the process proceeds to step S60. Otherwise, that is, if the size of the output data from the NN operation unit 102-k exceeds the memory capacity of the internal storage unit 104, the process proceeds to step S40. Furthermore, in the case where the number of divisions of the feature map has been set in the process of step S40 described later, which has been executed with respect to the upper-layer NN operation unit 102-(k-1) as the processing target, the determination in step S30 is made using the size of the output data from the NN operation unit 102-k determined by the divided feature map.

[0063] In step S40, it is determined to divide the feature map into two equal parts in the feature map division unit 101. When step S40 is executed, the size of the output data from the NN operation unit 102-k is calculated based on the data sizes of the respective regions after the feature map is divided, and the process returns to step S30. Thus, the set value of the number of divisions of the feature map is increased until the size of the output data from the NN operation unit 102-k when the feature map is divided into multiple regions becomes equal to or less than the memory capacity of the internal storage unit 104.

[0064] When the process proceeds from step S20 to step S50, in step S50, the storage location of the output data from the NN operation unit 102-k selected as the current processing target is determined to be the external storage device 13. When the process of step S50 is executed, the process proceeds to step S70.

[0065] When the process proceeds from step S30 to step S60, in step S60, the storage location of the output data from the NN operation unit 102-k selected as the current processing target is determined to be the internal storage unit 104. When the process of step S60 is executed, the process proceeds to step S70.

[0066] In step S70, it is determined whether k = N - 1. If k = N - 1, that is, when the NN operation unit 102-k selected as the current processing target is the intermediate layer immediately before the output layer (the case of the last stage of the intermediate layer), the Figure 6 processing shown in the flowchart ends. On the other hand, if it is not k = N - 1, the process proceeds to step S80.

[0067] In step S80, the value of k is incremented by 1, thereby advancing the NN arithmetic unit 102-k to be processed to the next layer. When the processing of step S80 is executed, the process returns to step S20, and the above-described process is repeated. Thus, each layer of the NN arithmetic units constituting the arithmetic processing unit 102 is sequentially selected as the processing target starting from the first-layer NN arithmetic unit 102-1 to determine the number of divisions of the feature map and the storage location of the intermediate data.

[0068] Furthermore, the method for determining the number of divisions of the feature map and the storage location of the intermediate data under the processing described above is merely an example. The number of divisions of the feature map and the storage location of the intermediate data can also be determined by other methods. For example, the number of divisions of the feature map and the number of layers of the neural network in which each layer of the NN arithmetic unit performs arithmetic processing, that is, the number of layers of the NN arithmetic unit of the arithmetic processing unit 102 that stores the intermediate data in the internal storage unit 104, can be determined respectively according to at least one of the following conditions. Figure 6

[0069] (Condition 1) The storage capacity of the internal storage unit 104

[0070] (Condition 2) The total arithmetic amount of the arithmetic processing performed by each layer of the NN arithmetic unit

[0071] (Condition 3) The data transfer bandwidth between the DNN arithmetic device 10 and the external storage device 13

[0072] (Condition 4) The amount of change in the data size before and after the arithmetic processing performed by each layer of the NN arithmetic unit

[0073] According to one embodiment of the present invention described above, the following operational effects are obtained.

[0074] (1) The DNN arithmetic device 10 is an information processing device that executes DNN arithmetic under a neural network composed of multiple layers. The DNN arithmetic device 10 performs arithmetic processing (NN arithmetic units 102-1 to 102-N of the arithmetic processing unit 102) corresponding to a specified layer of the neural network for each of the first region (e.g., region 31) and the second region (e.g., region 32) different from the first region in the feature map 30 input to the neural network. Subsequently, the result of the arithmetic processing for the first region is integrated with the result of the arithmetic processing for the second region and output as the result of the arithmetic processing for the feature map 30 (feature map integration unit 103). Therefore, in an information processing device that performs arithmetic using a neural network, it is possible to achieve high-speed processing without degrading the recognition accuracy.

[0075] ​(2) The DNN arithmetic unit 10 includes a feature map segmentation unit 101 that divides the feature map 30 into a first region and a second region. Therefore, the input feature map can be appropriately segmented.

[0076] (3) The feature map segmentation unit 101 divides the feature map 30 into a first region and a second region such that each of the first region and the second region includes redundant parts (for example, redundant parts 41 and 42 in regions 31 and 32) that repeat each other. Therefore, each NN arithmetic unit 102-1 to 102-N of the arithmetic processing unit 102 can accurately perform respective arithmetic processing for each of the segmented regions.

[0077] (4) The size of the redundant part is determined according to the size and stride of the filter used in the arithmetic processing performed by each NN arithmetic unit 102-1 to 102-N of the arithmetic processing unit 102. Therefore, when applying the filter to the boundary part of each of the segmented regions, the same result as that obtained when performing the operation on the feature map before segmentation can be obtained.

[0078] (5) The DNN arithmetic unit 10 includes NN arithmetic units 102-1 to 102-N, an internal storage unit 104, and a feature map integration unit 103. The NN arithmetic units 102-1 to 102-N are provided corresponding to each layer of the neural network and perform arithmetic processing for each of the first region and the second region. The internal storage unit 104 stores, at different times, the result of the arithmetic processing performed by the NN arithmetic unit 102-k corresponding to the k-th layer of the neural network for the first region and the result of the arithmetic processing performed by the NN arithmetic unit 102-k corresponding to the k-th layer of the neural network for the second region. The feature map integration unit 103 can integrate the result of the arithmetic processing performed by the NN arithmetic unit 102-(k+α) corresponding to the (k+α)-th layer of the neural network for the first region and the result of the arithmetic processing performed by the NN arithmetic unit 102-(k+α) corresponding to the (k+α)-th layer of the neural network for the second region. In this way, the arithmetic results of each region output from any intermediate layer among the intermediate layers provided between the input layer and the output layer in the arithmetic processing unit 102 can be integrated to perform DNN arithmetic.

[0079] (6) The result of the arithmetic processing integrated by the feature map integration unit 103 is stored in an external storage device 13 provided outside the DNN arithmetic unit 10. The result of the arithmetic processing stored in the external storage device 13 can be input to the NN arithmetic unit 102-(k+α+1) corresponding to the (k+α+1)-th layer of the neural network. In this way, since the integrated intermediate data can be used to perform the arithmetic processing of the remaining layers, the DNN arithmetic of the entire DNN arithmetic unit 10 can be continued.

[0080] (7) The NN operation unit 102-(k+α+1) corresponding to the (k+α+1)-th layer can perform convolution processing or pooling processing with a stride of 2 or more. In this way, when stored in the internal storage unit 104, the required memory capacity can be suppressed for the internal storage unit 104. In addition, when stored in the external storage device 13, the data transfer capacity can be suppressed for the data transfer band between the DNN operation device 10 and the external storage device 13.

[0081] (8) The DNN operation device 10 includes: a feature map division unit 101 that divides the feature map 30 into a plurality of regions 31 to 34 including at least a first region and a second region; NN operation units 102-1 to 102-N that are provided corresponding to each layer of the neural network and perform arithmetic processing on each of the regions 31 to 34; an internal storage unit 104 that stores the results of the arithmetic processing performed by the NN operation units 102-1 to 102-N; and a feature map integration unit 103 that integrates the results of the arithmetic processing respectively performed by the NN operation unit 102-k corresponding to a specified layer of the neural network on the regions 31 to 34 and stores them in the external storage device 13 provided outside the DNN operation device 10. The number of divisions of the feature map by the feature map division unit 101 and the number of layers of the NN operation units of the arithmetic processing unit 102 that stores the intermediate data in the internal storage unit 104 before the feature map integration unit 103 integrates the results of the arithmetic processing are determined according to at least any one of (Condition 1) the storage capacity of the internal storage unit 104, (Condition 2) the total arithmetic amount of the arithmetic processing performed by each layer of the NN operation unit, (Condition 3) the data transfer band between the DNN operation device 10 and the external storage device 13, and (Condition 4) the change amount of the data size before and after the arithmetic processing performed by each layer of the NN operation unit. Therefore, the number of divisions of the feature map in the feature map division unit 101 and the number of layers of the NN operation units of the arithmetic processing unit 102 that stores the intermediate data in the internal storage unit 104 can be appropriately determined respectively.

[0082] (9) The vehicle control device 1 includes a DNN operation device 10 and an action plan creation unit 15 that creates an action plan for the vehicle. The DNN operation device 10 performs DNN operations based on a feature map representing sensor information related to the surrounding conditions of the vehicle. The action plan creation unit 15 creates an action plan for the vehicle based on the result of the DNN operation output from the DNN operation device 10. Therefore, the action plan for the vehicle can be appropriately created using the result of the DNN operation performed by the DNN operation device 10.

[0083] Furthermore, in the embodiment described above, the DNN arithmetic unit 10 included in the in-vehicle control device 1 mounted in the vehicle has been described by taking, as an example, a device that performs DNN arithmetic based on sensor information related to the surrounding conditions of the vehicle to identify the surrounding conditions of the vehicle. However, the present invention is not limited thereto. The present invention can be applied to various information processing devices as long as they are information processing devices that perform DNN arithmetic under a neural network composed of multiple layers.

[0084] The above-described embodiments and various modifications are merely examples, and the present invention is not limited to these as long as the features of the invention are not impaired. In addition, each embodiment and various modifications can be adopted alone or combined arbitrarily. Furthermore, although the above describes various embodiments and modifications, the present invention is not limited to these. Other forms contemplated within the technical idea of the present invention are also included within the scope of the present invention.

[0085] Symbol Explanation

[0086] 1... In-vehicle control device, 2... Camera, 3... LiDAR, 4... Radar, 10... DNN arithmetic unit, 11... Sensor fusion unit, 12... Feature map storage unit, 13... External storage device, 15... Action plan formulation unit, 101... Feature map segmentation unit, 102... Arithmetic processing unit, 103... Feature map integration unit, 104... Internal storage unit, 121... Convolution processing unit, 122... Activation processing unit, 123... Pooling processing unit.

Claims

1. An information processing apparatus that performs DNN operations under a neural network composed of multiple layers, characterized in that, for each of a first region and a second region different from the first region in a feature map input to the neural network, an arithmetic processing corresponding to a prescribed layer of the neural network is performed, the result of the arithmetic processing for the first region is integrated with the result of the arithmetic processing for the second region, and output as the result of the arithmetic processing for the feature map, the information processing apparatus includes: a feature map segmentation unit that segments the feature map into a plurality of regions including at least the first region and the second region; an NN arithmetic unit that is provided corresponding to each layer of the neural network and performs the arithmetic processing for each of the plurality of regions; an internal storage unit that stores the result of the arithmetic processing performed by the NN arithmetic unit; and a feature map integration unit that integrates the results of the arithmetic processing respectively performed by the NN arithmetic unit corresponding to the prescribed layer of the neural network for the plurality of regions, and stores the integrated result in an external storage device provided outside the information processing apparatus, the number of segments of the feature map by the feature map segmentation unit and the number of layers of the neural network for which the NN arithmetic unit performs the arithmetic processing before the feature map integration unit integrates the results of the arithmetic processing are determined according to the storage capacity of the internal storage unit, the total arithmetic amount of the arithmetic processing performed by the NN arithmetic unit, the data transfer bandwidth between the information processing apparatus and the external storage device, and at least any one of the change amounts of the data sizes before and after the arithmetic processing performed by the NN arithmetic unit.

2. The information processing apparatus according to claim 1, characterized in that, it includes a feature map segmentation unit that segments the feature map into the first region and the second region.

3. The information processing apparatus according to claim 2, characterized in that, the feature map segmentation unit segments the feature map into the first region and the second region in such a manner that each of the first region and the second region includes a redundant portion that overlaps with each other.

4. The information processing apparatus according to claim 3, characterized in that, the size of the redundant portion is determined according to the size and stride of a filter used in the arithmetic processing.

5. The information processing apparatus according to claim 1, characterized in that, when the plurality of regions include only the first region and the second region, the information processing apparatus includes: an NN arithmetic unit that is provided corresponding to each layer of the neural network and performs the arithmetic processing for each of the first region and the second region; an internal storage unit that stores, at different times, the result of the arithmetic processing performed by the NN arithmetic unit corresponding to the k-th layer of the neural network for the first region and the result of the arithmetic processing performed by the NN arithmetic unit corresponding to the k-th layer of the neural network for the second region; and A feature map integration unit that integrates the result of the arithmetic processing performed by the NN arithmetic unit corresponding to the (k+α)-th layer of the neural network for the first region and the result of the arithmetic processing performed by the NN arithmetic unit corresponding to the (k+α)-th layer for the second region.

6. The information processing apparatus according to claim 5, wherein, the result of the arithmetic processing integrated by the feature map integration unit is stored in an external storage device provided outside the information processing apparatus, and the result of the arithmetic processing stored in the external storage device is input to the NN arithmetic unit corresponding to the (k+α+1)-th layer of the neural network.

7. The information processing apparatus according to claim 5, wherein, the NN arithmetic unit corresponding to the (k+α+1)-th layer performs a convolution process or a pooling process with a stride of 2 or more.

8. An information processing apparatus that performs DNN arithmetic under a neural network composed of multiple layers, wherein, it includes: a feature map segmentation unit that segments a feature map input to the neural network into a plurality of regions such that each of the segmented regions contains mutually redundant parts; an NN arithmetic unit that is provided corresponding to each layer of the neural network and performs a prescribed arithmetic process for each of the plurality of regions; an internal storage unit that stores the result of the arithmetic process performed by the NN arithmetic unit; and a feature map integration unit that integrates the results of the arithmetic processes respectively performed by the NN arithmetic unit corresponding to a prescribed layer of the neural network for the plurality of regions and stores the integrated result in an external storage device provided outside the information processing apparatus, the size of the redundant part is determined according to the size and stride of the filter used in the arithmetic process, the number of segments of the feature map by the feature map segmentation unit and the number of layers of the neural network for which the NN arithmetic unit performs the arithmetic process before the feature map integration unit integrates the results of the arithmetic process are determined according to the storage capacity of the internal storage unit, the total arithmetic amount of the arithmetic processes performed by the NN arithmetic unit, the data transfer bandwidth between the information processing apparatus and the external storage device, and at least any one of the change amounts of the data sizes before and after the arithmetic processes performed by the NN arithmetic unit.

9. A vehicle control apparatus, wherein, it includes: the information processing apparatus according to any one of claims 1 to 8; and an action plan formulation unit that formulates an action plan for the vehicle, the information processing apparatus performs the arithmetic process based on sensor information related to the surrounding conditions of the vehicle, and the action plan formulation unit formulates the action plan for the vehicle based on the result of the arithmetic process output from the information processing apparatus.

Images