Information processing device, information processing method, program, and computer readable recording medium

Abstract

To reduce a quantization error while suppressing increase in processing time.SOLUTION: In an information processing device (10), a storage device stores a neural network (42) and quantization parameters (31, 32). The neural network comprises: a specific intermediate layer (46a); and a quantization layer (46b) which quantizes data (50) as aggregate of values to be output by the specific intermediate layer on the basis of the quantization parameters. An arithmetic circuit executes: repetition processing for repeatedly executing arithmetic processing for obtaining an output result of the neural network by inputting image data in the neural network and storing the data to be output by the specific intermediate layer in the storage device; and change processing for changing the quantization parameters on the basis of distribution of the values in the data stored in the storage device in parallel with the arithmetic processing.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The technology disclosed in this specification relates to information processing using neural networks.

[0002] Patent Document 1 discloses a technique for quantizing data output from an intermediate layer of a neural network. In this technique, multiple pieces of calibration image data are input to a neural network that has undergone machine learning as a calibration process. This identifies the distribution of values ​​(specifically, the maximum and minimum values) in the data output from the intermediate layer, and determines quantization parameters (specifically, Z values ​​and S values) for quantizing the data based on the identified distribution. In the inference phase, quantization is performed based on the quantization parameters determined in the calibration process. Quantization can reduce the amount of calculation in the neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-144805 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1, quantization is performed during the inference phase using a quantization parameter determined in the calibration process. That is, the quantization parameter is fixed in the inference phase. Therefore, if an image with a different pixel value distribution from the calibration image data used in the calibration process is input in the inference phase, a large error will occur during quantization.

[0005] Also, a quantization technique called Dynamic Quantization is known. Dynamic Quantization calculates the distribution of data output from the intermediate layer in the inference phase, calculates a quantization parameter based on the calculated distribution, and performs quantization based on the calculated quantization parameter. This technique can reduce quantization errors. However, this technique lengthens the processing time of the neural network because the calculation of the quantization parameter and the processing of the quantization layer are performed serially for the output data of the intermediate layer.

[0006] This specification proposes a technique for reducing quantization errors while suppressing an increase in processing time. [Means for solving the problem]

[0007] The information processing device disclosed in this specification includes a storage device and an arithmetic circuit. The storage device stores a neural network and a quantization parameter. The neural network includes a specific hidden layer and a quantization layer that quantizes data, which is a collection of values ​​output by the specific hidden layer, based on the quantization parameter. The arithmetic circuit executes an iterative process and a modification process. In the iterative process, the arithmetic circuit inputs image data to the neural network to obtain an output result of the neural network, and repeatedly executes a computation process that stores the data output by the specific hidden layer in the storage device. In the modification process, the arithmetic circuit modifies the quantization parameter based on the distribution of values ​​in the data stored in the storage device, in parallel with the computation process.

[0008] In this information processing device, the arithmetic circuit executes a change process (i.e., a process of changing the quantization parameter) in parallel with the arithmetic process. In the change process, the arithmetic circuit changes the quantization parameter based on the distribution of values ​​in the data stored in the storage device (i.e., data previously output by the intermediate layer). Therefore, in the arithmetic process after the change process, quantization is performed according to the changed quantization parameter. In this way, the quantization parameter is changed according to the data output by the intermediate layer, so that the quantization error can be reduced. Furthermore, in this information processing device, the arithmetic circuit executes the change process in parallel with the arithmetic process, so that the time required for processing from the intermediate layer to the quantization layer is short. In other words, the processing time by the neural network is short. As described above, this information processing device can reduce the quantization error while suppressing an increase in processing time. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of an information processing device 10. [Figure 2] Block diagram of neural network 42. [Figure 3] 4 is a flowchart showing NN calculation processing and quantization parameter change processing according to the first embodiment. [Figure 4] 10 is a flowchart showing a first example of a process for determining whether a start condition is met. [Figure 5] 10 is a flowchart showing a second example of a process for determining whether a start condition is met. [Figure 6] 10 is a flowchart showing a third example of a process for determining whether a start condition is met. [Figure 7] 10 is a flowchart showing an NN calculation process, a similarity determination process, and a quantization parameter change process according to the second embodiment. [Figure 8] FIG. 10 is an explanatory diagram of image data used to determine similarity. [Figure 9] 10 is a histogram of pixel values ​​of pixel data in one example. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the information processing device described above, the arithmetic circuit may store, in the storage device, a plurality of pieces of data output by the specific intermediate layer in a plurality of the arithmetic processes, and the arithmetic circuit may change, in the change process, the quantization parameter based on a distribution of values ​​in the plurality of pieces of data stored in the storage device.

[0011] According to this configuration, the frequency of execution of the change process is reduced, which reduces the calculation load on the arithmetic circuit and enables the processing time to be shortened.

[0012] The information processing device may be mounted on a vehicle. The information processing device may further include a camera that captures an image of a scene ahead of the vehicle. The arithmetic circuit may input image data captured by the camera to the neural network during the arithmetic processing.

[0013] According to this configuration, the image of the area ahead of the vehicle can be analyzed by the neural network. Since the processing time of the neural network is short, the image of the area ahead of the vehicle can be analyzed in higher real time.

[0014] In the information processing device described above, the arithmetic circuit may determine the timing to execute the change process based on a travel distance of the vehicle.

[0015] According to this configuration, even if the input image data (that is, scenery, etc.) changes due to the running of the vehicle, the quantization error can be reduced by changing the quantization parameter.

[0016] In the information processing device described above, the arithmetic circuit may determine the timing to execute the change process based on position information of the vehicle.

[0017] According to this configuration, even if the input image data (that is, scenery, etc.) changes due to a change in the position of the vehicle, the quantization error can be reduced by changing the quantization parameter.

[0018] In the information processing device described above, the arithmetic circuit may determine the timing of executing the change process based on time.

[0019] In the information processing device, the arithmetic circuit may store the image data to be input to the neural network in the storage device, and the arithmetic circuit may identify at least one first image data and at least one second image data that was input to the neural network before the first image data from the image data stored in the storage device, and perform the change process when the similarity between the first image data and the second image data is lower than a reference value.

[0020] The above-mentioned "similarity" is an index indicating that the higher the degree of similarity, the more similar the first image data and the second image data are, and the lower the degree of similarity, the more dissimilar the first image data and the second image data are.

[0021] According to this configuration, when the input image data changes significantly (that is, when the similarity decreases), the quantization error can be reduced by changing the quantization parameter.

[0022] According to this configuration, even if the input image data (that is, scenery, etc.) changes over time, the quantization error can be reduced by changing the quantization parameter.

[0023] In the above-mentioned information processing device, the arithmetic circuit may identify the amount of change in the quantization parameter in the change process, and determine the timing of executing the change process so that the smaller the amount of change, the longer the execution interval until the next change process.

[0024] According to this configuration, when the amount of change in the quantization parameter is small, the frequency of execution of the change process can be reduced, thereby reducing the calculation load on the arithmetic circuit.

[0025] Furthermore, the processing executed by the information processing device described above can also be realized as an information processing method, a program, or a computer-readable recording medium on which a program is recorded.

[0026] (First embodiment) An information processing device 10 of the first embodiment shown in FIG. 1 is mounted on a vehicle such as an automobile or a motorcycle. The information processing device 10 captures an image of the area ahead of the vehicle and detects objects (e.g., pedestrians, other vehicles, obstacles, etc.) from the captured image. The information processing device 10 transmits the detection results to an output device 90 mounted on the vehicle. For example, the output device 90 may be a display device. In this case, the output device 90 can display the position and type of the object identified by the information processing device 10. Furthermore, in a vehicle with a driving assistance function or an autonomous vehicle, the output device 90 may be a device that controls the traveling of the vehicle (e.g., a device that controls acceleration / deceleration or a device that controls a steering angle). In this case, the output device 90 can control the speed and steering angle of the vehicle according to the detection results of the information processing device 10.

[0027] The information processing device 10 has a computer 11 and a camera 16. The computer 11, the camera 16, and the output device 90 are connected to each other via a data bus 18 provided in the vehicle. The camera 16 captures an image of the area ahead of the vehicle. The computer 11 has an arithmetic circuit 12 and a storage device 14. The storage device 14 has a non-volatile memory and may further have a volatile memory. The storage device 14 stores an object detection program 40 and quantization parameters 31 and 32. The quantization parameters 31 and 32 are values ​​defined as variables in the object detection program 40, and can also be considered as part of the object detection program 40. The object detection program 40 has a neural network 42. The arithmetic circuit 12 is configured by a CPU or the like and performs various calculations. The arithmetic circuit 12 can execute the object detection program 40.

[0028] The neural network 42 is a machine-learned neural network. FIG. 2 shows the configuration of the neural network 42. As shown in FIG. 2, the neural network 42 has multiple hidden layers 46 between an input layer 44 and an output layer 48. Image data captured by the camera 16 is input to the input layer 44. In the first half of the neural network 42, convolutional layers 46a and pooling layers 46c are alternately arranged to extract features from the image. In the second half of the neural network 42, fully connected layers 46d and activation functions 46e are alternately arranged to identify objects in the image. Note that any activation function (e.g., a step function, a sigmoid function, a tanh function, a ReLU function, a softmax function, etc.) can be used as the activation function 46e. The output layer 48 outputs the type and position (i.e., the position in the image) of the object identified in the image as a detection result. The neural network 42 also has a quantization layer 46b. The quantization layer 46b quantizes the data output by the first convolutional layer 46a.

[0029] When the vehicle starts, the arithmetic circuit 12 executes the object detection program 40. Then, the arithmetic circuit 12 captures an image of the area ahead of the vehicle using the camera 16, and executes neural network arithmetic processing (hereinafter referred to as NN arithmetic processing) on ​​the captured image data. The NN arithmetic processing is a so-called inference process. In the NN arithmetic processing, the arithmetic circuit 12 inputs the image data to the neural network 42, performs calculations according to each intermediate layer 46, and outputs a detection result from the output layer 48. The arithmetic circuit 12 repeatedly executes the NN arithmetic processing while the vehicle is traveling, and detects objects ahead of the vehicle in real time.

[0030] The flowchart on the right side of FIG. 3 shows the processing executed in the convolution layer 46a and the quantization layer 46b of the neural network 42. In the convolution layer 46a, the arithmetic circuit 12 applies a filter to image data input from the previous layer and outputs a feature map 50. The feature map 50 is image data and is a collection (i.e., a tensor) of pixel values ​​(i.e., scalar values). Each pixel value of the feature map 50 is represented as an FP32 (i.e., a 32-bit floating-point number). The feature map 50 is input to the quantization layer 46b. In the quantization layer 46b, the arithmetic circuit 12 quantizes the feature map 50 based on one of the quantization parameters 31 and 32. More specifically, in the quantization layer 46b, the arithmetic circuit 12 converts each pixel value of the feature map 50 from FP32 to INT8 (i.e., an 8-bit integer) based on one of the quantization parameters 31 and 32. In the quantization layer 46b, each pixel value may be converted from FP32 to INT16, INT4, etc. Which of the quantization parameters 31 and 32 is used will be described in detail later. Each of the quantization parameters 31 and 32 has a Z value and an S value. In the quantization layer 46b, the arithmetic circuit 12 converts the FP32 value x into an INT8 value q using the following Equation 1: q=Z+round(x / S) (Equation 1) The round function in the above formula 1 is a function that rounds the value x / S to an integer, for example, a rounding function. As is clear from the above formula 1, the S value is a value that represents the range when rounding a numerical value. Furthermore, the Z value is a value that determines the zero point of the value q. In this way, in the quantization layer 46b, the arithmetic circuit 12 converts each pixel value of the feature map 50 from FP32 to INT8 and outputs a feature map 50a having each converted pixel value. By quantizing the feature map 50 in the quantization layer 46b in this way, the calculation load in the subsequent NN calculation processing is reduced. As a result, the arithmetic circuit 12 can perform the NN calculation processing at high speed.

[0031] Furthermore, the arithmetic circuit 12 stores the feature map 50 output by the convolutional layer 46a in the storage device 14. As described above, the arithmetic circuit 12 repeatedly executes the NN arithmetic process. Each time the arithmetic circuit 12 executes the NN arithmetic process, the arithmetic circuit 12 stores the feature map 50 output by the convolutional layer 46a in the storage device 14. Therefore, the storage device 14 stores multiple feature maps 50 output in past NN arithmetic processes. Note that, since there is an upper limit to the storage capacity, the storage device 14 stores a maximum of the feature maps 50 from the past N times.

[0032] While repeating the NN arithmetic process, the arithmetic circuit 12 executes the quantization parameter change process in parallel with the NN arithmetic process. As described above, the quantization layer 46b uses either the quantization parameters 31 or 32. Immediately after starting the vehicle, the quantization parameters 31 and 32 (i.e., the S value and Z value) may be set to predetermined values ​​or may be set to values ​​used the previous time the vehicle was used. Either the quantization parameters 31 or 32 may be used first, but the quantization parameter change process will be described below using the case where the quantization parameter 31 is used first as an example.

[0033] When a predetermined start condition is satisfied while repeatedly executing the NN arithmetic processing, the arithmetic circuit 12 executes the quantization parameter change processing shown in FIG. 3 . The start condition for the quantization parameter change processing will be described in detail later. When the quantization parameter change processing starts, the arithmetic circuit 12 reads the feature map 50 from the storage device 14 in step S2. Here, the arithmetic circuit 12 reads N feature maps 50 stored in the storage device 14 in the past N NN arithmetic processing operations. The arithmetic circuit 12 calculates parameters indicating the distribution of each pixel value (hereinafter referred to as distribution parameters) based on each pixel value of the read feature maps 50. For example, the arithmetic circuit 12 calculates the maximum value Max and minimum value Min of each pixel value of the multiple feature maps 50 as the distribution parameters. Note that the maximum value Max and minimum value Min may be calculated directly from each pixel value, or may be calculated by removing outliers from each pixel value. Alternatively, the average value and variance of each pixel value may be calculated from each pixel value, and the maximum value Max and minimum value Min may be calculated based on the average value and variance.

[0034] Next, in step S4, the arithmetic circuit 12 calculates an S value and a Z value as quantization parameters from the distribution parameters of each pixel value. For example, the S value can be calculated from the following Equation 2. S = (Max - Min) / 255 (Equation 2) Furthermore, for example, the Z value can be calculated from the following Equation 3. Z=round(-(Max+Min) / 2S) (Equation 3) Furthermore, when the value q is an unsigned integer, the Z value can be calculated from the following Equation 4 instead of Equation 3 above. Z=round(-Min / S) (Equation 4)

[0035] Next, in step S6, the arithmetic circuit 12 rewrites the quantization parameters 31 and 32, whichever is not currently being used, with the quantization parameter calculated in step S4. Because the quantization parameter 31 is being used in the NN arithmetic process during the first quantization parameter change process, the arithmetic circuit 12 rewrites the quantization parameter 32 with the value calculated in step S4. Next, in step S8, the arithmetic circuit 12 switches the quantization parameter used in the NN arithmetic process. Because the quantization parameter 31 is being used in the NN arithmetic process during the first quantization parameter change process, the arithmetic circuit 12 switches the quantization parameter to be used from the quantization parameter 31 to the quantization parameter 32. As a result, the quantization parameter used in the quantization layer 46b is changed to the value newly calculated in step S6. Therefore, when the quantization layer 46b is executed after step S8 is executed, quantization is performed based on the newly calculated quantization parameter. As described above, the quantization parameter used in the quantization layer 46b is changed by the quantization parameter change process.

[0036] When the start condition for the quantization parameter change process is satisfied again after the execution of the quantization parameter change process, the arithmetic circuit 12 executes the quantization parameter change process again. The arithmetic circuit 12 calculates the quantization parameter in steps S2 and S4. Since the quantization parameter 32 is used in the NN arithmetic process during the execution of the second quantization parameter change process, the arithmetic circuit 12 rewrites the quantization parameter 31 to a new value in step S6. Thereafter, in step S8, the arithmetic circuit 12 switches the quantization parameter used in the NN arithmetic process from the quantization parameter 32 to the quantization parameter 31 (i.e., the newly calculated quantization parameter). Therefore, thereafter, quantization is performed based on the newly calculated quantization parameter.

[0037] As described above, in the quantization parameter change process, the arithmetic circuit 12 calculates a new quantization parameter based on each pixel value of the feature map 50 for N iterations stored in the storage device 14, and changes the quantization parameter used in the NN arithmetic process to a new value. By repeatedly executing the quantization parameter change process, the quantization parameter used in the NN arithmetic process is updated to a value that matches the distribution of each pixel value of the most recent feature map. Because the image data input to the neural network 42 is an image of the area in front of the vehicle, the distribution of each pixel value of the image data input to the neural network 42 rarely changes suddenly in a short period of time. Therefore, the distribution of each pixel value of the feature map 50 rarely changes suddenly in a short period of time. For this reason, by updating the quantization parameter to match the pixel value of the most recent feature map 50, it is possible to reduce quantization errors that occur in the quantization layer 46b.

[0038] Next, an example of a start condition for the quantization parameter change process will be described.

[0039] (Example 1) The start condition of the first example is that the vehicle's travel distance reaches a predetermined value. FIG. 4 shows the process of determining the start condition of the first example. While repeating the NN calculation process, the calculation circuit 12 executes the process of FIG. 4. In step S12, the calculation circuit 12 calculates the distance traveled by the vehicle since the previous execution of the quantization parameter change process from the detection value of the vehicle's odometer. Next, in step S14, the calculation circuit 12 determines whether the travel distance calculated in step S12 exceeds A km. The calculation circuit 12 repeats steps S12 and S14 until step S14 determines YES. If step S14 determines YES, the calculation circuit 12 executes the quantization parameter change process in step S16.

[0040] When the start condition in the first example is used, the quantization parameter change process is executed every time the vehicle travels A km. Since the scenery ahead of the vehicle changes more easily as the vehicle travels longer, the quantization parameter change process is executed according to the vehicle travel distance, thereby making it possible to appropriately reduce the quantization error.

[0041] (Example 2) The start condition in the second example is that the vehicle position has changed. FIG. 5 shows the process of determining the start condition in the second example. In the second example, the vehicle position at the time of the previous execution of the quantization parameter change process is stored in the storage device 14 or the like. In step S22, the arithmetic circuit 12 acquires the current position of the vehicle from a GPS device mounted on the vehicle and calculates the straight-line distance between the current position of the vehicle and the position at which the previous quantization parameter change process was executed. Next, in step S24, the arithmetic circuit 12 determines whether the straight-line distance calculated in step S22 exceeds B km. The arithmetic circuit 12 repeats steps S22 and S24 until step S24 returns YES. If step S24 returns YES, the arithmetic circuit 12 executes the quantization parameter change process in step S26, and then stores the vehicle position at the time of the execution in step S28.

[0042] When the second example start condition is used, the quantization parameter change process is executed when the vehicle's driving area changes significantly. Since the scenery ahead of the vehicle is likely to change when the driving area changes, executing the quantization parameter change process according to the driving area makes it possible to appropriately reduce the quantization error.

[0043] In addition, when the start condition is set based on GPS information as in the second example, the calculation circuit 12 may identify area information (e.g., urban area, suburban area, mountainous area, etc.) of the current location of the vehicle based on map data from the car navigation system, and execute the quantization parameter change process when the area information changes.

[0044] (Example 3) The start condition in the third example is that a predetermined time has elapsed. FIG. 6 shows the process of determining the start condition in the third example. In the third example, the execution time of the previous quantization parameter change process is stored in the storage device 14 or the like. In step S32, the arithmetic circuit 12 calculates the difference between the execution time of the previous quantization parameter change process and the current time as the elapsed time. Next, in step S34, the arithmetic circuit 12 determines whether the elapsed time calculated in step S32 exceeds C time. The arithmetic circuit 12 repeats steps S32 and S34 until step S34 returns YES. If step S34 returns YES, the arithmetic circuit 12 executes the quantization parameter change process in step S36, and then stores the execution time in step S38.

[0045] When the start condition of the third example is used, the quantization parameter change process is executed when time has passed. As time passes, the scenery changes not only as the vehicle travels, but also as the time of day (for example, morning, noon, night) changes. Therefore, by executing the quantization parameter change process according to the elapsed time, the quantization error can be appropriately reduced. Note that the quantization parameter change process may be executed based on the time instead of the elapsed time.

[0046] Furthermore, in the first to third examples, the thresholds determining the timing of executing the quantization parameter change process (i.e., values ​​A, B, and C shown in FIGS. 4 to 6) were fixed values. However, the thresholds may be changed according to the amount of change in the quantization parameter (i.e., S value, Z value) in the quantization parameter change process. More specifically, the thresholds may be changed so that the smaller the amount of change in the quantization parameter, the longer the interval until the next quantization parameter change process. For example, in the first example of FIG. 4, the amount of change in the quantization parameter may be calculated in step S16, and the smaller the calculated amount of change, the larger the value X. For example, in the second example of FIG. 5, the amount of change in the quantization parameter may be calculated in step S26, and the smaller the calculated amount of change, the larger the value B. For example, in the third example of FIG. 6, the amount of change in the quantization parameter may be calculated in step S36, and the smaller the calculated amount of change, the larger the value C. If the amount of change in the quantization parameter is small, it is likely that the vehicle's driving environment will continue to change only slightly thereafter. Therefore, if the amount of change in the quantization parameter is small, the quantization error will not increase significantly even if the interval until the next quantization parameter change process is lengthened. Furthermore, by lengthening the execution interval until the next quantization parameter change process in this way, it is possible to reduce the calculation load on the arithmetic circuit 12. Therefore, the NN arithmetic process can be performed at higher speed in the arithmetic circuit 12.

[0047] The start conditions of the first to third examples may be combined. Also, the start conditions of the first to third examples may be combined with other conditions.

[0048] In the information processing device 10 described above, N feature maps 50 are stored in the storage device 14, and the quantization parameter change process is performed based on each pixel value of the stored feature maps 50. That is, the quantization parameter change process is performed less frequently than the NN operation process. This configuration reduces the computational load on the arithmetic circuit 12. Furthermore, because the quantization parameters are changed based on N feature maps 50, the quantization parameters calculated in the quantization parameter change process are stable even if the scenery ahead of the vehicle changes temporarily. However, in other embodiments, the quantization parameter change process may be performed once for each NN operation process. That is, after the feature map 50 is stored in the storage device 14 in the NN operation process, the NN operation process and the quantization parameter change process may be performed in parallel, and the quantization parameter for the next NN operation process may be changed by the quantization parameter change process. In this case, in the quantization parameter change process, a new quantization parameter is calculated based on one feature map 50. Even with this configuration, the quantization parameter change processing and the NN arithmetic processing are executed in parallel, so the NN arithmetic processing can be executed in a shorter time than when the quantization parameter change processing is executed in the middle of the NN arithmetic processing.

[0049] (Second embodiment) FIG. 7 shows the NN operation process and the quantization parameter change process of the second embodiment. In the NN operation process of the second embodiment, the operation circuit 12 stores image data input to the input layer 44 (i.e., image data captured by the camera 16) in the storage device 14. The storage device 14 accumulates m pieces of image data input in the past m NN operation processes. FIG. 8 shows m pieces of image data G stored in the storage device 14. Note that the number attached to the image data G indicates the recency of the image data, with image data G1 being the image input to the input layer 44 in the most recent NN operation process, and image data Gm being the image data input to the input layer 44 in the NN operation process m-1 times prior. Except for storing the image data G in the storage device 14, the NN operation process of the second embodiment is the same as the NN operation process of the first embodiment.

[0050] In the second embodiment, the arithmetic circuit 12 periodically executes the similarity determination process (steps S20 and S22 in FIG. 7). For example, the arithmetic circuit 12 executes the similarity determination process every time it executes the NN arithmetic process P times (P is an integer). In the similarity determination process, the arithmetic circuit 12 calculates the similarity between the most recently input Q pieces of first image data Ga and the Q pieces of second image data Gb input before that, and determines whether or not to execute the quantization parameter change process (i.e., steps S2 to S8) depending on the similarity. Note that FIG. 8 shows a case where Q=4 as an example. The similarity determination process will be described in detail below.

[0051] In step S20, the arithmetic circuit 12 identifies first image data Ga and second image data Gb from among the multiple image data G stored in the storage device 14. The arithmetic circuit 12 identifies the latest Q image data G (four image data G1 to G4 in FIG. 8) as the first image data Ga. The arithmetic circuit 12 also identifies the Q image data G immediately before the first image data Ga (four image data G5 to G8 in FIG. 8) as the second image data Gb.

[0052] Furthermore, in step S20, the arithmetic circuit 12 determines the distribution of pixel values ​​of the first image data Ga and the second image data Gb. A pixel value is a value assigned to each pixel and indicates the brightness of the pixel. A pixel value is assigned to each of RGB (i.e., red, green, and blue). The pixel value is expressed by 256 levels ranging from 0 to 255. FIG. 9 illustrates the distribution of pixel values ​​for each of RGB of the first image data Ga. The horizontal axis of FIG. 9 indicates the pixel value (i.e., a value from 0 to 255). The vertical axis of FIG. 9 indicates the number of pixels having a corresponding pixel value (i.e., the frequency of occurrence of pixels having a corresponding pixel value). Note that the vertical axis of FIG. 9 indicates the sum of all the numbers of pixels in the Q pieces of first image data Ga. As shown in FIG. 9, the arithmetic circuit 12 calculates a histogram for each of RGB for the first image data Ga. Similarly, the arithmetic circuit 12 calculates a histogram for each of RGB for the second image data Gb.

[0053] Furthermore, in step S20, the arithmetic circuit 12 calculates the similarity between the first image data Ga and the second image data Gb based on the histogram of the first image data Ga and the histogram of the second image data Gb. As an index indicating the similarity, various indices such as a correlation coefficient, KL divergence, and three-dimensional Euclidean distance can be used. After calculating the similarity, the arithmetic circuit 12 then determines in step S22 whether the similarity is lower than a reference value, and determines whether to execute a quantization parameter change process depending on the determination result.

[0054] (When using correlation coefficients) When using a correlation coefficient, the arithmetic circuit 12 calculates the correlation coefficient D using the following formulas 1 and 2. The arithmetic circuit 12 calculates the correlation coefficient D for each of RGB. Hereinafter, the correlation coefficients D for RGB will be referred to as correlation coefficients Dr, Dg, and Db, respectively.

number

number

[0055] (When using KL divergence) When KL divergence is used, the arithmetic circuit 12 calculates the KL divergence DKL using the following equation 3. The arithmetic circuit 12 calculates the KL divergence DKL for each of RGB. Hereinafter, the RGB KL divergences DKL will be referred to as KL divergences DKLr, DKLg, and DKLb, respectively.

number

[0056] (When using 3D Euclidean distance) When using three-dimensional Euclidean distance, the arithmetic circuit 12 calculates the average value of the pixel values ​​of the first image data Ga and the average value of the pixel values ​​of the second image data Gb. The arithmetic circuit 12 calculates the average value of the pixel values ​​for each of RGB. Hereinafter, the average values ​​of the pixel values ​​of each of RGB in the first image data Ga will be referred to as Ar, Ag, and Ab, and the average values ​​of the pixel values ​​of each of RGB in the second image data Gb will be referred to as Br, Bg, and Bb. Next, the arithmetic circuit 12 calculates the three-dimensional Euclidean distance X using the following equation 4:

number

[0057] As described above, the arithmetic circuit 12 calculates the similarity between the most recently captured first image data Ga and the second image data Gb captured earlier, and executes the quantization parameter change process only when the similarity is low. Therefore, the quantization parameter change process is executed when the scenery ahead of the vehicle changes significantly. This allows the quantization error to be appropriately reduced. Furthermore, the quantization parameter change process is not executed when the change in the scenery ahead of the vehicle is small and there is no need to change the quantization parameter, so the frequency with which the quantization parameter change process is executed can be reduced.

[0058] Furthermore, pixel values ​​of image data captured by the camera 16 while the vehicle is traveling may temporarily change significantly. For example, if headlights from an oncoming vehicle directly hit the camera 16, the pixel values ​​of the image data may increase at that moment, but immediately thereafter, the pixel values ​​may return to their normal values. Performing a quantization parameter change process when pixel values ​​suddenly change in this way may result in the quantization parameter not being set appropriately. In contrast, the second embodiment can appropriately process sudden changes in pixel values. For example, consider a case where a sudden change in pixel values ​​causes the pixel value of only the image data G2 in FIG. 8 to increase. In this case, the arithmetic circuit 12 determines whether to change the quantization parameter based on the similarity between the four pieces of first image data Ga (i.e., G1 to G4) and the four pieces of second image data Gb (i.e., G5 to G8). Even if the pixel value of only the image data G2 increases, if the change in pixel values ​​of the first image data Ga as a whole is small, the arithmetic circuit 12 determines that the similarity between the first image data Ga and the second image data Gb is high, and does not perform a quantization parameter change process. In this way, according to the second embodiment, when a sudden change in pixel value occurs, it is possible to suppress changes in the quantization parameter, thereby preventing the quantization parameter from being changed to an inappropriate value.

[0059] Furthermore, in the second embodiment, the similarity determination process (ie, steps S20 and S22) and the quantization parameter change process (ie, steps S2 to S8) are executed in parallel with the NN calculation process, so that the speed of the NN calculation process can be increased.

[0060] Although FIG. 8 illustrates an example in which Q=4, the integer Q can be set to any value. Similarly, the integer P (i.e., the execution frequency of the similarity determination process) can be set to any value. For example, Q may be greater than P, or Q may be smaller than P. Furthermore, P and Q may both be 1. When P is 1, the similarity determination process is executed once for each execution of the NN calculation process. When Q is 1, there is one each of the first image data Ga and the second image data Gb. Furthermore, while FIG. 8 illustrates the first image data Ga and the second image data Gb as being continuous in time series, there may be a time interval between the first image data Ga and the second image data Gb. For example, the first image data Ga may be composed of image data G1 to G4, and the second image data Gb may be composed of image data G7 to G10.

[0061] In addition, in FIG. 8, the image data immediately before the first image data Ga is used as the second image data Gb, but the image data used when the previous quantization parameter change process was executed may be used as the second image data Gb.

[0062] The structure of the neural network 42 shown in FIG. 2 is merely an example. The neural network 42 may have any number of intermediate layers 46, and any type of layer may be arranged as each intermediate layer 46. In the above-described embodiment, the output data of the convolutional layer is input to the quantization layer, but the data input to the quantization layer may be the output data of any intermediate layer. For example, the output data of a pooling layer, a fully connected layer, or an activation function may be input to the quantization layer. In this case, the output data of the intermediate layer before the quantization layer is stored in the storage device 14, and the quantization parameter change process can be performed based on the distribution of values ​​in the output data.

[0063] Furthermore, although the information processing device 10 of the above-described embodiment analyzes image data of the area in front of the vehicle using a neural network, the technology disclosed herein may also be applied to the analysis of other image data. For example, the technology disclosed herein may be applied when other vehicles (such as airplanes and ships) or drones capture images of the area in front of or around the vehicle and repeatedly input the image data into a neural network. The technology disclosed herein may also be applied when each frame of a video is repeatedly input into a neural network. Note that the technology disclosed herein can reduce the amount of calculation in the arithmetic circuit and speed up NN arithmetic processing, making the technology disclosed herein particularly useful for standalone devices (such as in-vehicle devices) with limited processing power in the arithmetic circuit.

[0064] Furthermore, in the above-described embodiment, the maximum value Max and minimum value Min are identified from each pixel value of the feature map 50, and the S value and Z value, which are quantization parameters, are calculated based on the maximum value Max and minimum value Min. However, the quantization parameter may be calculated in any way as long as it can be calculated from the distribution of values. For example, the quantization parameter may be calculated based on an index representing the distribution other than the maximum value Max and minimum value Min. Furthermore, values ​​other than the S value and the Z value may be used as the quantization parameter.

[0065] Although the embodiments have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples exemplified above. The technical elements described in this specification or drawings exhibit technical utility alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. Furthermore, the technology exemplified in this specification or drawings simultaneously achieves multiple objectives, and achieving one of these objectives itself has technical utility. [Explanation of symbols]

[0066] 12: arithmetic circuit, 14: storage device, 16: camera, 46a: convolution layer, 46b: quantization layer, 50: feature map

Claims

1. An information processing device (10) having a storage device (14) and an arithmetic circuit (12), The storage device stores a neural network (42) and quantization parameters (31, 32), The neural network A specific intermediate layer (46a); a quantization layer (46b) that quantizes data (50) that is a collection of values ​​output by the specific intermediate layer based on the quantization parameter; and The arithmetic circuit a repetitive process of repeatedly executing a calculation process in which image data is input to the neural network to obtain an output result of the neural network and the data output by the specific intermediate layer is stored in the storage device; a change process of changing the quantization parameter based on a distribution of values ​​in the data stored in the storage device, in parallel with the calculation process; To execute Information processing device.

2. the arithmetic circuit accumulates the plurality of pieces of data output by the specific intermediate layer in the plurality of arithmetic processes in the storage device; the arithmetic circuit changes the quantization parameter based on a distribution of values ​​in the plurality of pieces of data stored in the storage device in the change process; The information processing device according to claim 1 .

3. The information processing device is mounted on a vehicle, The vehicle further includes a camera (16) for capturing an image of a scene ahead of the vehicle; the arithmetic circuit inputs image data captured by the camera into the neural network in the arithmetic processing; 3. The information processing device according to claim 1.

4. The information processing device according to claim 3 , wherein the arithmetic circuit determines the timing to execute the change process based on a travel distance of the vehicle.

5. The information processing device according to claim 3 , wherein the arithmetic circuit determines the timing to execute the change process based on position information of the vehicle.

6. In the arithmetic processing, the arithmetic circuit stores the image data input to the neural network in the storage device; the arithmetic circuit identifies at least one first image data and at least one second image data input to the neural network before the first image data from the image data stored in the storage device, and executes the change process when the similarity between the first image data and the second image data is lower than a reference value. The information processing device according to claim 1 .

7. The information processing device according to claim 1 , wherein the arithmetic circuit determines the timing of executing the change process based on time.

8. 2. The information processing device according to claim 1, wherein the arithmetic circuit specifies a change amount of the quantization parameter in the change process, and determines a timing for executing the change process such that the smaller the change amount, the longer the execution interval until the next change process.

9. An information processing method executed by an information processing device having a storage device, the storage device stores a neural network and a quantization parameter; The neural network The specific middle class and a quantization layer that quantizes data that is a collection of values ​​output by the specific intermediate layer based on the quantization parameter; and a repetitive process of repeatedly executing a calculation process in which image data is input to the neural network to obtain an output result of the neural network and the data output by the specific intermediate layer is stored in the storage device; a change process of changing the quantization parameter based on a distribution of values ​​in the data stored in the storage device, in parallel with the calculation process; An information processing method comprising:

10. A program having a neural network, The neural network The specific middle class and a quantization layer that quantizes data that is a collection of values ​​output by the specific intermediate layer based on a quantization parameter; and The program is provided to an information processing device. a repetitive process of repeatedly executing a calculation process in which image data is input to the neural network to obtain an output result of the neural network and the data output by the specific intermediate layer is stored in a storage device of the information processing device; a change process of changing the quantization parameter based on a distribution of values ​​in the data stored in the storage device, in parallel with the calculation process; Execute program.

11. A computer-readable recording medium on which the program according to claim 10 is recorded.

Images