Inference device and inference method

Abstract

To reduce power consumption by reducing the amount of convolution calculations. [Solution] The inference device has an input unit that inputs multiple pieces of image data that are continuous in a predetermined direction, a thinning processing unit that performs thinning processing on each of the multiple pieces of image data input by the input unit so that pixel arrangements that show multiple patterns are repeated and so that pixels in the spatial direction do not overlap, and outputs multiple pieces of thinned image data, a convolution calculation unit that applies multiple thinning filters that constitute the multiple patterns divided from a filter having a weighting coefficient to each of the multiple pieces of thinned image data to perform a convolution calculation, an inference unit that performs inference on the multiple pieces of image data using a time series filter based on the multiple convolution calculation results by the convolution calculation unit, and an output unit that outputs the inference results by the inference unit.

Description

[Technical Field]

[0001] The present invention relates to an inference device and an inference method. [Background technology]

[0002] The image processing capacity required for Artificial Intelligence (AI) processing in automotive and industrial Internet of Things (IoT) applications is expanding. For example, AI processing, which was previously performed on High Definition (HD) cameras in conventional applications, now requires processing from multiple 4K resolution cameras, which increases the power consumption of computer resources performing AI processing.

[0003] While the processing performance required for AI is improving, edge applications require low-power computing resources and AI chips that meet fanless requirements. It is difficult to process high-resolution images with AI while meeting these power consumption requirements. Conventionally, methods for reducing power consumption in AI processing have often been used, focusing on inference processing and making AI models lighter through pruning and quantization.

[0004] Non-Patent Document 1 discloses various types of pruning methods for convolutional neural networks (CNNs). Non-Patent Document 2 discloses a technique for reducing the amount of memory required for multiplication processing and storing weights by optimizing the number of bits used for CNN weights. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Yang He, Lingao Xiao, “Structured Pruning for Deep Convolutional Neural Networks: A survey,” IEEE trans. PAMI, 2023. [Non-patent document 2] Wakana Nogami, et. al., “Optimizing Weight Value Quantization for CNN Inference,” IJCNN, 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the conventional technologies mentioned above, it has become difficult to achieve the performance required for recent AI processing, and methods for further improving power efficiency are desired.

[0007] An object of the present invention is to reduce power consumption by reducing the amount of convolution calculation. [Means for solving the problem]

[0008] An inference device according to one aspect of the invention disclosed in the present application is characterized by having an input unit that inputs a plurality of image data that are continuous in a predetermined direction; a thinning processing unit that performs a thinning process on each of the plurality of image data input by the input unit so that pixel arrangements that show a plurality of patterns are repeated and so that pixels in the spatial direction do not overlap, and outputs a plurality of thinned image data; a convolution calculation unit that applies a plurality of thinning filters that constitute the plurality of patterns divided from a filter having a weighting coefficient to each of the plurality of thinned image data to perform a convolution calculation; an inference unit that performs inference on the plurality of image data using a time series filter based on the plurality of convolution calculation results by the convolution calculation unit; and an output unit that outputs the inference results by the inference unit.

[0009] Another aspect of the invention disclosed in the present application is an inference device comprising: an input unit that inputs a plurality of image data that are continuous in a predetermined direction; a thinning processing unit that performs a thinning process on each of the plurality of image data input by the input unit so that pixel arrangements that show a plurality of patterns are repeated and so that pixels in the spatial direction do not overlap, and outputs a plurality of thinned image data; a convolution operation unit that synthesizes the plurality of thinned image data thinned by the thinning processing unit to generate composite thinned image data, and applies a filter having a weighting coefficient to the composite thinned image data to perform a convolution operation; an inference unit that performs inference on the plurality of image data using a time series filter based on the convolution operation result by the convolution operation unit; and an output unit that outputs the inference result by the inference unit. [Effects of the Invention]

[0010] According to the exemplary embodiment of the present invention, it is possible to reduce the amount of convolution calculations, thereby achieving low power consumption. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an inference device. [Figure 2] FIG. 2 is a block diagram showing an example of the functional configuration of the inference device. [Figure 3] FIG. 3 is an explanatory diagram showing an example of the configuration of a convolutional neural network. [Figure 4] FIG. 4 is an explanatory diagram showing an example of convolution processing in a convolutional neural network. [Figure 5] FIG. 5 is an explanatory diagram showing a second example of convolution processing in a convolutional neural network. [Figure 6] FIG. 6 is an explanatory diagram showing an example of thinning out input image data at a thinning rate of 1 / 2. [Figure 7]FIG. 7 is an explanatory diagram showing an example of applying a filter to thinned image data at a thinning rate of 1 / 2. [Figure 8] FIG. 8 is an explanatory diagram showing an example of thinning out input image data at a thinning rate of 1 / 3. [Figure 9] FIG. 9 is an explanatory diagram showing an example of CNN filter division at a thinning rate of 1 / 3. [Figure 10] FIG. 10 is an explanatory diagram showing an example of applying a CNN filter to thinned image data at a thinning rate of 1 / 3. [Figure 11] FIG. 11 is an explanatory diagram showing a third example of convolution processing at a thinning rate of 1 / 2. [Figure 12] FIG. 12 is an explanatory diagram showing a third example of convolution processing at a thinning rate of 1 / 3. [Figure 13] FIG. 13 is an explanatory diagram showing an example of switching the operation of the signal processing unit 202. In FIG. [Figure 14] FIG. 14 is an explanatory diagram showing an example of controlling the synthesis weight in the time direction in the thinned-out CNN processing. [Figure 15] FIG. 15 is an explanatory diagram showing an example of time-series filter synthesis in thinned-out CNN processing. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments are examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. The present invention can also be implemented in various other forms. When there are multiple components having the same or similar functions, they may be described using the same reference numerals with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0013] In the embodiments, there may be a case where a process performed by executing a program is described. Here, a computer executes the program using a processor (e.g., a CPU or a GPU), and performs the process defined by the program while using storage resources (e.g., a memory) and interface devices (e.g., a communication port). Therefore, the entity that executes the program and performs the process may be the processor. Similarly, the entity that executes the program and performs the process may be a controller, device, system, computer, or node that has a processor.

[0014] In addition, in the embodiment, in addition to the general-purpose processor, if a deep neural network (DNN) processing unit is provided as an accelerator for performing specific processing at high speed, the DNN processing, which has a heavy computation load, is performed by the DNN processor. In this case, by performing signal processing taking into account the utilization rate of the parallel computing unit, it is possible to further improve power efficiency.

[0015] <Figure 1 Example of hardware configuration of an inference device> 1 is a block diagram showing an example of the hardware configuration of an inference device 100. The inference device 100 includes a DNN processor 101, a general-purpose processor 102, a main memory 103, and an I / O interface 104.

[0016] The DNN processor 101 is hardware specialized for DNN processing. The DNN processor 101 is limited in the signal processing it can perform compared to the general-purpose processor 102. The general-purpose processor 102 executes general-purpose processing that the DNN processor 101 cannot perform. Because the DNN processor 101 is determined to perform only specific processing, the general-purpose processor 102 is unnecessary when general-purpose processing is not required. On the other hand, when emphasis is placed on the performance of general-purpose processing, the DNN processor 101 is unnecessary.

[0017] The I / O interface 104 receives input of the time-series image data group 110 from the camera, receives input of distance information of the subject from the camera or a sensor (e.g., LiDAR) not shown, and outputs the execution results of the DNN processor 101 and the general-purpose processor 102 and the time-series image data group 110 to the external device 120. The time-series image data group 110 is a series of digital image data in a time sequence, such as video data. The external device 120 displays the execution results of the DNN processor 101 and the general-purpose processor 102 and the time-series image data group 110.

[0018] The DNN processor 101 includes a multiply-accumulate calculator 111 that performs matrix multiply-accumulate (convolution) operations at high speed, a vector calculator 112 that performs specific pipeline processing such as pipelining the activation function of the DNN, an accumulator for normalization, and division, a local memory 113, a control unit 114, and an I / O interface 115.

[0019] The I / O interface 115 inputs data to the DNN processor 101 and outputs data calculated by the DNN processor 101. The local memory 113 stores the time-series image data group 110 input to the DNN processor 101 and intermediate results of the DNN processing. The product-sum calculator 111 and vector calculator 112 read and write data from the local memory 113 and perform as much calculation processing as possible using the time-series image data group 110 and intermediate results of the DNN processing. The control unit 114 controls and generates addresses to read and write from the local memory 113 according to the thinning rate and thinning pattern, and inputs only necessary data to the product-sum calculator 111 and vector calculator 112. The control unit 114 also sets time constants for the product-sum calculator 111 and vector calculator 112 when combining probabilities.

[0020] This reduces the frequency of access between the main memory 103 and the DNN processor 101 of the inference device 100 and the data size of the data stored in the main memory 103, improving the power efficiency of the inference device 100. Therefore, a method of reducing the time-series image data group 110 input to the DNN processor 101 used in one processing or the intermediate results of the DNN processing becomes effective.

[0021] In the following, unless otherwise specified, a case will be described in which both the DNN processor 101 and the general-purpose processor 102 are included.

[0022] <Figure 2: Example of functional configuration of the inference device 100> 2 is a block diagram showing an example of the functional configuration of the inference device 100. The inference device 100 has an input unit 201, a signal processing unit 202, and an output unit 203. The input unit 201 inputs the time-series image data group 110 and weighting coefficients of the CNN filter. The input unit 201 passes the time-series input image data and weighting coefficients that make up the time-series image data group 110 to the signal processing unit 202.

[0023] The output unit 203 outputs the execution results of the DNN processor 101 and the general-purpose processor 102 and the time-series image data group 110 to the external device 120. The input unit 201 and the output unit 203 are configured by the I / O interface 104 shown in FIG. 1. The signal processing unit 202 processes the time-series image data group 110 to perform AI inference. The signal processing unit 202 is configured by the DNN processor 101, the general-purpose processor 102, and the main memory 103.

[0024] The signal processing unit 202 includes a thinning unit 221, a convolution operation unit 222, an inference unit 223, and a control unit 224. The thinning unit 221 performs thinning processing on each of a plurality of input image data that are consecutive in the time direction, thinning out the input image data so that pixel arrays showing a plurality of patterns are repeated and so that pixels do not overlap in the space direction, and outputs a plurality of thinned image data. Such thinning processing is defined by a thinning rate and a thinning pattern (described later in Convolution Processing Example 2 and Convolution Processing Example 3). The thinning unit 221 may also determine the number of thinned image data pieces at different times using a time constant, or may combine these thinned image data (described later in Convolution Processing Example 2 and Convolution Processing Example 3).

[0025] The convolution operation unit 222 performs a convolution operation on the thinned image data from the thinning processing unit 221 using a CNN filter to which a weighting coefficient is set, and outputs the convolution operation result to the inference unit 223. The CNN filter is divided by a thinning pattern (described later in convolution processing example 2).

[0026] The inference unit 223 combines the convolution calculation results from the convolution calculation unit 222 in the time direction using a time series filter, and outputs the inference result. The time series filter is, for example, a Kalman filter or a particle filter.

[0027] The control unit 224 controls the time constant, thinning rate, and thinning pattern applied to the thinning processing unit 221. The control unit 224 also controls the time constant applied to the inference unit 223. For example, if the inference device 100 is an on-board device, the control by the control unit 224 is performed based on vehicle and road traffic information such as the vehicle speed, weather, map information, and scene information, as well as pre-set information.

[0028] For example, when the control unit 224 determines from the vehicle speed and map information that the vehicle is traveling on a highway, the main task is to detect a distant vehicle ahead. Therefore, the control unit 224 sets the thinning rate lower than the thinning rate when traveling on an ordinary road. This prevents deterioration in the detection accuracy of small objects. Furthermore, when the control unit 224 determines from the vehicle speed and map information that the vehicle is traveling on a highway, a quick brake response is required, so the time constant is set lower than the time constant when traveling on an ordinary road. The time constant and the thinning rate are set in this manner.

[0029] <Figure 3: Example of a convolutional neural network configuration> FIG. 3 is an explanatory diagram showing an example configuration of a convolutional neural network. The convolutional neural network 300 has a structure in which various processing layers, such as an input layer 301, a convolutional layer 302, a normalization layer 303, a pooling layer 304, a probability layer 305, an activation layer 306, and a fully connected layer 307, are stacked in multiple stages within a DNN processor 101. Among these, the DNN processor 101 generally executes processes that involve multiply-and-accumulate operations on a large amount of weights, input data, and features. Processing of layers that the DNN processor 101 cannot handle is executed by the general-purpose processor 102. In general, the processing of the convolutional layer 302 in the convolutional neural network 300 requires a large amount of computation, and efficient processing of the convolutional layer 302 can improve the power efficiency of the entire inference process.

[0030] <Figure 4 Example 1 of convolution processing in the convolutional neural network 300> 4 is an explanatory diagram showing an example of convolution processing in a convolutional neural network 300. In the convolutional neural network 300, a product-sum calculator 111 performs a convolution operation on input image data I(t) at time t using a CNN filter F with a kernel size of 3×3. The result of the convolution operation is output to the next layer as a feature C. Finally, after processing within the convolutional neural network 300, various inference results Pn(t) at time t, such as regression, classification, and object recognition, are output.

[0031] <Figure 5 Example 2 of convolution processing in the convolutional neural network 300> 5 is an explanatory diagram showing a second example of convolution processing in the convolutional neural network 300. The second example of convolution processing illustrates a thinning convolution process. The thinning convolution process utilizes the fact that the input image data I(t) has a correlation in the time direction to thin out pixels in the input image data I(t) according to a thinning rate.

[0032] The thinning rate is the frequency at which pixels are thinned out in the input image data I(t). If the thinning rate is 1 / p (p is an integer greater than or equal to 2), one pixel out of p consecutive pixels in the spatial direction (row direction, column direction) and the temporal direction is thinned out in both the spatial and temporal directions. In FIG. 5, the thinning rate is 1 / 2, so the input image data I(t) is thinned out pixel by pixel in both the spatial and temporal directions to become thinned-out image data J(t). In FIG. 5, the input image data I(t) is assumed to be 6 × 6 pixels.

[0033] In the thinning convolution process, the CNN filter 400 is also thinned out pixel by pixel at a thinning rate of 1 / 2 to produce a first CNN filter F1 and a second CNN filter F2. Combining the first CNN filter F1 and the second CNN filter F2 results in the original CNN filter F.

[0034] The product-sum calculator 111 alternately applies the first CNN filter F1 and the second CNN filter F2 to the thinned image data J(t), and outputs the convolution calculation result as a feature 510. That is, the first CNN filter F1 and the second CNN filter F2 are applied periodically based on the thinning rate (once every two times if the thinning rate is 1 / 2). The feature C(t), which is the convolution calculation result, is output to the next layer. Therefore, if the thinning rate is 1 / p, the divided p CNN filters F1, F2, ..., Fp are applied once every p times.

[0035] After processing within the convolutional neural network 300, various inference results Pn(t) at time t, such as regression, classification, and object recognition, are output. Then, the inference device 100 outputs the inference result Pn(t|t-1, t-2, ...) by synthesizing the inference results Pn(t), Pn(t-1), Pn(t-2), ... at times t, t-1, t-2, ... using the time series filter 500. This reduces the overall amount of calculation by the inference device 100 and suppresses deterioration in accuracy.

[0036] [Figure 6 Example of thinning out input image data I(t) to I(t-3) at a thinning rate of 1 / 2] Fig. 6 is an explanatory diagram showing an example of thinning out the input image data I(t) to I(t-3) at a thinning rate of 1 / 2. Fig. 6 shows an example of thinning out the input image data I(t) to I(t-3) at a thinning rate of 1 / 2. The thinned image data J(t) to J(t-3) is image data obtained by thinning out the input image data I(t) to I(t-3) at a thinning rate of 1 / 2.

[0037] When there is no need to distinguish between the input image data I(t) to I(t-3), they are referred to as input image data I. When there is no need to distinguish between the thinned image data J(t) to J(t-3), they are referred to as thinned image data J.

[0038] When the thinning rate is 1 / 2, the thinned image data J(t) and the thinned image data J(t-2) are the same image data, and the thinned image data J(t-1) and the thinned image data J(t-3) are the same image data.

[0039] [Figure 7 Example of applying a CNN filter to thinned image data J(t) and J(t-1) with a tatami thinning rate of 1 / 2] FIG. 7 is an explanatory diagram showing an example of applying filters to thinned image data J(t) and J(t-1) at a thinning rate of 1 / 2. For the thinned image data J(t), the first CNN filter F1 and the second CNN filter F2 are applied alternately. The stride of the first CNN filter F1 and the second CNN filter F2 is 2. As a result, the feature C(t) is calculated as the result of the convolution operation.

[0040] For the thinned image data J(t-1), the second CNN filter F2 and the first CNN filter F1 are applied alternately in this order. The stride of the first CNN filter F1 and the second CNN filter F2 is 2. As a result, the feature 510(t-1) is calculated as the result of the convolution operation.

[0041] The weights used in convolution processing example 2 become part of the kernel weights, and when regular thinning is performed at a thinning rate of 1 / 2, the kernels that are slid and multiplied to the input image data I are essentially divided into two types, the first CNN filter F1 and the second CNN filter F2, and the number of multiplications when outputting pixels of each feature amount is reduced. In calculations using the time series filter 500 after convolution, for example, when the probability of object recognition is output as the inference result Pn(t), the time series filter 500 is used to calculate the conditional probabilities for previous times t-1, t-2, ..., and thereby combine the inference results Pn(t-1), Pn(t-2), ... with the inference result Pn(t).

[0042] When the change in the time direction of the input image data I is sufficiently small, a large gain can be obtained by increasing the time constant of the time series filter 500, but when the change in the time direction is large, accuracy deteriorates. Therefore, the time constant that determines the time range of the inference result Pn to be synthesized from the inference results Pn(t), Pn(t-1), Pn(t-2), ... must be appropriately selected for each scene.

[0043] In general, by keeping the time constant sufficiently small, it is possible to obtain a minimum synthesis gain while suppressing the possibility of accuracy degradation. If it is predicted that the time direction of the input image data I from the camera will change little based on the vehicle speed, information from external sensors, and road traffic information, the accuracy can be improved by increasing the synthesis gain by increasing the time constant.

[0044] The determination of the time range of the inference result Pn to be combined using a time constant may be done by simply combining data from time t to n times, or by using an exponential decay function to effectively reduce the influence of past times distant from time t. For example, if the combined inference result at times t, t-1, and t-2 is Y(t), and the inference results before combination at times t, t-1, and t-2 are X(t), X(t-1), and X(t-2), respectively, the implicit determination of the time range of the inference result Pn to be combined using the time constant τ is expressed by the following equation (1).

[0045] Y(t)=X(t)+X(t-1)×exp(-1 / τ)+X(t-2)×exp(-2 / τ) ···(1)

[0046] [Figure 8 Example of thinning out input image data I(t) to I(t-3) at a thinning rate of 1 / 3] FIG. 8 is an explanatory diagram showing an example of thinning out input image data I(t) to I(t-3) at a thinning rate of 1 / 3. When the thinning rate is 1 / 3, the input image data I(t) to I(t-3) are thinned out to produce thinned image data K(t) to K(t-3). The thinned image data K(t) to K(t-2) are image data in which different pixels are thinned out. Furthermore, the thinned image data K(t) and the thinned image data K(t-3) are the same image data.

[0047] [Figure 9 Example of CNN filter division with a thinning rate of 1 / 3] 9 is an explanatory diagram showing an example of CNN filter division at a thinning rate of 1 / 3. When the thinning rate is 1 / 3, a 4x4 CNN filter G is divided into a first CNN filter G1, a second CNN filter G2, and a third CNN filter G3. When the first CNN filter G1, the second CNN filter G2, and the third CNN filter G3 are combined, the original CNN filter G is obtained.

[0048] [Figure 10 Example of applying a CNN filter to thinned image data K(t), K(t-1), and K(t-2) with a tatami thinning rate of 1 / 3] FIG. 10 is an explanatory diagram showing an example of applying CNN filters to thinned image data K(t), K(t-1), and K(t-2) at a thinning rate of 1 / 3. The stride of the first CNN filter G1, the second CNN filter G2, and the third CNN filter G3 is 3. For the thinned image data K(t), the first CNN filter G1, the second CNN filter G2, and the third CNN filter G3 are applied in this order. As a result, the feature amount D(t) is calculated as the result of the convolution operation.

[0049] For thinned image data K(t-1), the third CNN filter G3, the first CNN filter G1, and the second CNN filter G2 are applied in this order. As a result, the feature D(t-1) is calculated as the convolution operation result. For thinned image data K(t-2), the second CNN filter G2, the third CNN filter G3, and the first CNN filter G1 are applied in this order. As a result, the feature D(t-2) is calculated as the convolution operation result.

[0050] In the convolution processing example 2, the thinning rates are 1 / 2 and 1 / 3, but the thinning rate can be other than 1 / 2 and 1 / 3. The thinning pattern may be any pattern as long as fluctuations in image degradation are acceptable.

[0051] Furthermore, the thinning process can be performed not on the input image data I(t) and I(t-1) but on their feature quantities C(t) and C(t-1).

[0052] Furthermore, the input image data I(t), I(t-1) and their feature quantities C(t), C(t-1) also have a dimension in the channel direction. Therefore, in the channel direction, as in the time direction, pixel thinning and synthesis using the time series filter 500 can be applied. In particular, the channel direction is resistant to positional deviation because pixels surrounding the pixel of interest are convolved, and it is easy to tolerate fluctuations in degradation when thinning is performed using an arbitrary thinning pattern.

[0053] <Convolution processing example 3 of convolutional neural network 300> Next, we will explain Example 3 of convolution processing of the convolutional neural network 300. Generally, accelerators for neural networks achieve high efficiency by operating a large number of arithmetic units simultaneously in parallel. Generally, such hardware achieves maximum efficiency in dense matrix operations. Therefore, skipping some multiplication processes of the kernels of the CNN filters F and G often requires special instruction overhead or a hardware mechanism, and hardware that does not have such a mechanism may not effectively achieve the effect of reducing the amount of calculation.

[0054] In the convolution processing example 3, the CNN filters F and G are used as they are without being divided into multiple filters. Therefore, compared to the convolution processing example 2, the amount of calculation is reduced.

[0055] [Figure 11 Example 3 of convolution processing with a thinning rate of 1 / 2] 11 is an explanatory diagram showing a third example of convolution processing at a thinning rate of 1 / 2. The inference device 100 generates thinned image data J(t) and J(t-1) from multiple input image data I(t) and I(t-1) at times t and t-1. The inference device 100 combines the thinned image data J(t) and J(t-1) to generate combined thinned image data J(t, t-1).

[0056] As in FIG. 4, the product-sum calculator 111 performs a convolution operation on the combined and thinned image data J(t, t-1) using the CNN filter F to generate a combined feature C(t, t-1) that combines the features C(t) and C(t-1) at times t and t-1. The feature C(t, t-1) is output to the next layer. After processing within the convolutional neural network 300, various inference results Pn(t) at time t and Pn(t-1) at time t-1, such as regression, classification, and object recognition, are output. The inference device 100 then combines the inference results Pn(t) and Pn(t-1) at times t and t-1 using the time-series filter 500 to output the inference result Pn(t|t-1).

[0057] [Figure 12 Example 3 of convolution processing with a thinning rate of 1 / 3] 12 is an explanatory diagram showing a third example of convolution processing with a thinning rate of 1 / 3. The inference device 100 generates thinned image data K(t), K(t-1), and K(t-2) from input image data I(t), I(t-1), and I(t-2) at multiple times t, t-1, and t-2. The inference device 100 combines the thinned image data K(t), K(t-1), and K(t-2) to generate combined thinned image data K(t, t-1, t-2).

[0058] As in FIG. 4, the product-sum calculator 111 performs a convolution operation on the combined and thinned image data K(t, t-1, t-2) using the CNN filter G to generate a composite feature D(t, t-1, t-2) that combines the features D(t), D(t-1), and D(t-2) at times t, t-1, and t-2. The feature C(t, t-1, t-2) is output to the next layer. After processing within the convolutional neural network 300, various inference results Pn(t) at time t, Pn(t-1) at time t-1, and Pn(t-2) at time t-2, such as regression, classification, and object recognition, are output. Then, the inference device 100 outputs the inference result Pn(t|t-1, t-2) by synthesizing the inference results Pn(t), Pn(t-1), and Pn(t-2) at times t, t-1, and t-2 using the time series filter 500.

[0059] In this way, in convolution processing example 3, the inference device 100 combines thinned image data from multiple time points, performs calculations with all weights of the kernels of the CNN filters F and G, and outputs composite features C(t, t-1) and D(t, t-1, t-2) that combine features from multiple time points, thereby improving hardware utilization efficiency through processing that is essentially equivalent to dense matrix calculations.As a result, the data transfer size and transfer frequency of the main memory 103 and the DNN processor 101 can be reduced, and improved power efficiency can be expected.

[0060] In the convolution processing example 3, the thinning rates are 1 / 2 and 1 / 3, but other thinning rates are also applicable. The pattern of thinned pixels may be any pattern as long as variations in image degradation are tolerable. In particular, the channel direction is resistant to positional deviations because it is convoluted with the pixels surrounding the pixel of interest, and it is easy to tolerate variations in degradation when thinning with any pattern.

[0061] Furthermore, the thinning process can be performed not on the input image data I(t), I(t-1), I(t-2) but on their feature quantities D(t), D(t-1), D(t-2).

[0062] In addition, the input image data I(t), I(t-1), and I(t-2) and their feature quantities D(t), D(t-1), and D(t-2) also have a dimension in the channel direction. Therefore, pixel thinning can be applied to the channel direction as well, just as in the time direction.

[0063] <Fig. 13: Example of operation switching of the signal processing unit 202> 13 is an explanatory diagram showing an example of operation switching of the signal processing unit 202. Depending on the type of inference processing, the control unit 224 switches between a codec 1301, a normal CNN process 1310 in which the convolution operation unit 222 executes a convolution operation without thinning out the input image data I, and a first thinned-out CNN process 1311 and a second thinned-out CNN process 1312 in which the thinned-out input image data I is thinned out by the thinning unit 221 and then the convolution operation is executed by the convolution operation unit 222. The first thinned-out CNN process 1311 and the second thinned-out CNN process 1312 have different thinning rates and time constants.

[0064] Specifically, for example, the control unit 224 accepts a selection of one of log data storage processing, segmentation processing, long-distance object recognition processing, and short-distance object detection processing as the inference processing type.

[0065] When the control unit 224 receives a selection of the log data storage process as the inference process type, the control unit 224 selects the codec 1301 and controls the signal processing unit 202 to output the log data to the codec 1301. Since the log data storage process requires the log data itself, the control unit 224 controls the log data to be sent to the codec 1301 instead of the thinning unit 221 of the input image data I. The codec 1301 may be implemented in the DNN processor 101, or may be realized by causing the general-purpose processor 102 to execute a program stored in the main memory 103.

[0066] When the control unit 224 receives a selection of segmentation processing as the inference processing type, the control unit 224 selects the image processing unit 1300 and the DNN processor 101, executes image resizing 1302 on the input image data I in the image processing unit 1300, and controls the resized input image data I to undergo normal CNN processing 1302 (convolution operation in the product-sum calculator 111 without thinning processing). The DNN processor 101 outputs the normal inference result. This reduces the amount of calculation compared to when image resizing 1302 is not executed.

[0067] When the control unit 224 receives a selection of long-distance object recognition processing from object recognition as the inference processing type, the control unit 224 selects the image processing unit 1300 and the DNN processor 101, executes image cropping 1303 of a long-distance object from input image data I in the image processing unit 1300, and controls the DNN processor 101 to execute first thinning CNN processing 1311 and filtering of the cropped portion of input image data I by the time-series filter 500. As a result, a first inference result is output.

[0068] When the control unit 224 receives a selection of short-distance object detection processing from object recognition as the inference processing type, the control unit 224 selects the image processing unit 1300 and the DNN processor 101, executes image resizing 1304 for detecting short-distance objects from the input image data I in the image processing unit 1300, and controls the DNN processor 101 to execute second thinning CNN processing 1312 and filtering of the resized input image data I by the time series filter 500. This outputs a second inference result.

[0069] The control unit 224 accepts a selection of any one of segmentation processing, long-distance object recognition processing, and short-distance object detection processing as the inference processing type, but may also accept a resolution level as an inference processing type.

[0070] For example, when the control unit 224 receives a selection of low resolution indicating a resolution lower than a predetermined resolution and segmentation processing, the control unit 224 selects and executes image resizing 1302 and normal CNN processing 1310.

[0071] In addition, when the control unit 224 receives a selection of high resolution indicating a predetermined resolution or higher and object recognition, it selects image clipping 1303 and causes the image processing unit 1300 to perform image clipping 1303 of the target area in the input image data I.

[0072] In this case, if the subject distance of the target object is equal to or greater than a predetermined distance, control unit 224 selects long-distance object recognition processing. In this case, the moving speed of the object in the target area cut out by image cutout 1303 will be equal to or relatively slower than the moving speed of the moving object implementing inference apparatus 100. Therefore, control unit 224 controls time series filter 500 to increase the time constant.

[0073] If the subject distance of the target object is less than a predetermined distance, the control unit 224 selects short-distance object detection processing. In this case, the movement speed of the object in the target area cut out by image cutout 1303 is equal to or relatively faster than the movement speed of the moving body in which the inference device 100 is implemented. If the movement speed is relatively faster, the image of the object will become larger, so the control unit 224 selects image resize 1304 and executes image resize 1304 of the target area cut out by image cutout 1303. The control unit 224 then controls the time constant of the time series filter 500 to be smaller.

[0074] In this way, the operation of the signal processing unit 202 can be switched, reducing the amount of calculation and improving the object recognition accuracy.

[0075] <Figure 14 Example of time-domain synthesis weight control in thinning CNN processing> FIG. 14 is an explanatory diagram showing an example of time-domain synthesis weight control in thinning-out CNN processing. The intermediate layer in FIG. 14 is an arbitrary neural network layer. The probability calculation layer is a layer that calculates the probability for each class, such as a softmax layer in a typical DNN. The probability synthesis layer calculates synthesis weights using parameters from at least one of the results of the previous layer (see formula (1) above and formulas (2) and (3) below).

[0076] The thinned CNN processing is the first thinned CNN processing 1311 and the second thinned CNN processing 1312 shown in FIG. 13. The inference unit 223 reduces noise from the information source of the input image data I from the sensor or camera, as well as noise added by the thinning processing. To achieve this, the inference unit 223 weights and adds the input data used for synthesis in the time series filter using the estimated results of each noise, improving the power ratio between the true value and the noise. If the inference results of the CNN used for synthesis are X1 and X2, and the estimated noises are σ1 and σ2, the synthesis result Y is expressed by the following equation (2):

[0077] Y=X1×f(σ1)+X2×f(σ2)...(2)

[0078] In the above equation (2), the function f() is a function that determines the weight from the estimated noise. If it is assumed that the noise superimposed on X1 and X2 is uncorrelated, the function f() may be, for example, the inverse of the noise power. In this case, the synthesis result Y is expressed by the following equation (3).

[0079] Y=X1×(1 / σ1) / (1 / σ1+1 / σ2)+X2×(1 / σ2) / (1 / σ1+1 / σ2)...(3)

[0080] In the above formulas (2) and (3), σ1 and σ2 are one-dimensional scalar quantities, but the weights may be determined as a covariance matrix by assuming that the function f() is a multidimensional Gaussian distribution and substituting σ1 and σ2 into the function f(). If the covariance matrix of the noise of X1 is S1 and the covariance matrix of the noise of X2 is S2, the synthesis result Y is expressed by the following formula (4).

[0081] Y=X1×S1 -1 / (S1 -1 +S2 -1 )+X2×S2 -1 / (S1 -1 +S2 -1 ) ···(4)

[0082] The estimated values ​​of the noise power and noise covariance matrix in the above equations (2), (3), and (4) are found by calculating the variance and covariance matrix using some or multiple feature matrices from the input layer to the intermediate layer of the convolutional neural network 300 for X1 and X2 used in synthesis. The final synthesis result Y includes the reduction of the influence of the passage of time due to the time constant in addition to the weights.

[0083] <Figure 15 Example of time-series filter synthesis in thinning CNN processing> 15 is an explanatory diagram showing an example of time-series filter synthesis in thinned CNN processing. The convolution operation unit 222 may output information that can be tracked over time, such as the position or type on the image. In this case, the inference unit 223 synthesizes a series of convolution operation results output in time series from the convolution operation unit 222 using a linear filter such as a Kalman filter in time series. In this case, the point used as the synthesis input is one of the layers after the final layer of the intermediate layers.

[0084] The inference unit 223 uses the output result of one of the layers as an observation value and outputs the current state, which is the synthesis result of the time series filter. The current state is generated by synthesizing a predicted value, which predicts the current state from the state one time before, with the observed value. The inference device 100 synthesizes the predicted value and the observed value, for example, using a Kalman filter. In a Kalman filter, the uncertainty of the predicted value and the uncertainty of the observed value are expressed as covariance, and the predicted value and the observed value are synthesized using their reciprocal. The coefficients used in the synthesis are optimal filter coefficients known as Kalman gains. The updated current time is used for the predicted value for the next time. By sequentially repeating this process and synthesizing time series signals, noise in the time direction is reduced.

[0085] As described above, according to this embodiment, by performing convolution operations on image data that has been thinned out alternately in the continuous time direction, it is possible to reduce the amount of calculations performed by the neural network and thereby reduce power consumption, and by synthesizing the inference results using a time-series filter, it is possible to suppress deterioration in accuracy. In this way, by reducing the load of convolution processing, it is possible to execute highly efficient inference processing.

[0086] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0087] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0088] Information such as programs, tables, and files that realize each function can be stored in storage devices such as memory, hard disks, and SSDs (Solid State Drives), or on recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0089] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines that are necessary for implementation. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0090] 100 Reasoning device 201 Input section 202 Signal Processing Unit 203 Output section 221 Thinning out unit 222 Convolution Calculation Unit 223 Reasoning part 224 Control Unit 400 CNN filters 500 Time Series Filters 1300 Image Processing Unit

Claims

1. an input unit for inputting a plurality of image data items that are continuous in a predetermined direction; a thinning processing unit that performs a thinning process on each of the plurality of image data input by the input unit so that pixel arrangements showing a plurality of patterns are repeated and so that pixels do not overlap in a spatial direction, and outputs a plurality of thinned image data; a convolution calculation unit that applies a plurality of thinning filters that constitute the plurality of patterns divided from a filter having a weighting coefficient to each of the plurality of thinned image data to perform a convolution calculation; an inference unit that performs inference on the plurality of image data using a time series filter based on the plurality of convolution calculation results by the convolution calculation unit; an output unit that outputs an inference result by the inference unit; An inference device comprising:

2. an input unit for inputting a plurality of image data items that are continuous in a predetermined direction; a thinning processing unit that performs a thinning process on each of the plurality of image data input by the input unit so that pixel arrangements showing a plurality of patterns are repeated and pixels in a spatial direction do not overlap, and outputs a plurality of thinned image data; a convolution operation unit that generates composite thinned image data by synthesizing the plurality of thinned image data that have been thinned by the thinning processing unit, and applies a filter having a weighting coefficient to the composite thinned image data to perform a convolution operation; an inference unit that performs inference on the plurality of image data using a time series filter based on a result of the convolution operation by the convolution operation unit; an output unit that outputs an inference result by the inference unit; An inference device comprising:

3. 3. The inference device according to claim 1, the predetermined direction is a time direction, An inference device characterized by:

4. 3. The inference device according to claim 1, the predetermined direction is a channel direction; An inference device characterized by:

5. 3. The inference device according to claim 1, a control unit that controls the plurality of patterns; An inference device comprising:

6. 6. The inference device according to claim 5, The control unit controls the number of the plurality of patterns. An inference device characterized by:

7. 6. The inference device according to claim 5, the control unit controls the plurality of patterns based on a distance to a subject in the image data. An inference device characterized by:

8. 6. The inference device according to claim 5, The control unit controls the plurality of patterns when object recognition is selected, and controls not to execute the thinning process when a process other than the object recognition is selected. An inference device characterized by:

9. 9. The inference device according to claim 8, an image processing unit that processes the image data; the control unit controls the image processing unit to cut out an image area of ​​the subject from the image data if the distance to the subject in the image data is equal to or greater than a predetermined distance; the thinning processing unit performs the thinning processing on a plurality of image regions of the subject cut out from each of the plurality of image data by the image processing unit; An inference device characterized by:

10. 9. The inference device according to claim 8, an image processing unit that processes the image data; the control unit controls the image processing unit to reduce the image data if the distance to the subject in the image data is less than a predetermined distance; the thinning processing unit performs the thinning processing on the plurality of image data after the image processing unit reduces each of the plurality of image data. An inference device characterized by:

11. an input process for inputting a plurality of image data pieces that are continuous in a predetermined direction; a thinning process for thinning each of the plurality of image data input by the input process so that pixel arrangements showing a plurality of patterns are repeated and pixels in the spatial direction do not overlap, and outputting a plurality of thinned image data; a convolution operation process for applying a plurality of thinning filters constituting the plurality of patterns divided from a filter having a weighting coefficient to each of the plurality of thinned image data to perform a convolution operation; an inference process for performing an inference regarding the plurality of image data using a time series filter based on the plurality of convolution operation results obtained by the convolution operation process; an output process for outputting an inference result obtained by the inference process; 10. An inference method comprising:

Images