Information processing device

Abstract

To optimize, in an information processing device that executes a sparse convolution process, a power amount supplied to a processor in accordance with process the load on the processor.SOLUTION: An information processing device 100, 100b that executes sparse convolution processes on image data obtained in a time-series manner by an image sensor 10, includes; a processor (2) that executes arithmetic calculation of the sparse convolution process; a size predicting unit (5) that predicts a rule book size in the sparse convolution process, using a data rate of the image data obtained by the image sensor; and a power control unit (6) that outputs, to the processor, a control signal to control a power amount supplied to the processor in accordance with the prediction result of the size predicting unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device. [Background technology]

[0002] Sparse convolution is known as an image processing technique using a convolution neural network (CNN) (see, for example, Non-Patent Document 1). In sparse convolution, a rulebook is generated that links together elements of input image data, elements of a kernel (filter) used in convolution, and elements of data output by convolution, and an NPU (Neural Network Processing Unit) executes sparse convolution processing according to the rulebook. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] Zhiliang Zhou, "How does sparse convolution work?", [Retrieved August 9, 2022], Internet [URL: https: / / towardsdatascience.com / how-does-sparse-convolution-work-3257a0a8fd1] Summary of the Invention [Problem to be solved by the invention]

[0004] When an information processing device that performs sparse convolution processing processes data acquired over time by sensors such as event cameras, the data rate input to the NPU fluctuates, which in turn causes the size of the rulebook to be processed by the NPU to fluctuate. This fluctuation in the processing load of the NPU makes it difficult to optimize the amount of power supplied to the NPU. Specifically, if the amount of power supplied is set assuming the maximum load of the NPU, excess power will be generated when the load applied to the NPU is small. On the other hand, if the amount of power supplied is set assuming the average load, a power shortage will occur when the load applied to the NPU becomes greater than the average load. [Means for solving the problem]

[0005] The present disclosure can be realized in the following forms.

[0006] According to one embodiment of the present disclosure, there is provided an information processing device (100, 100b) that performs sparse convolution processing on image data acquired in time series by an image sensor (10). The information processing device includes a processor (2) that executes the calculation of the sparse convolution processing, a size prediction unit (5) that predicts a rulebook size for the sparse convolution processing using a data rate of the image data acquired by the image sensor, and a power amount control unit (6) that outputs a control signal to the processor to control the amount of power supplied to the processor according to a result of the prediction by the size prediction unit.

[0007] In this form of information processing device, the size prediction unit predicts the rulebook size for the sparse convolution processing using the data rate of the image data acquired by the image sensor, and the power amount control unit outputs a control signal to the processor for controlling the amount of power supplied to the processor in accordance with the result of the prediction by the size prediction unit, thereby making it possible to optimize the amount of power supplied to the processor in accordance with the amount of calculation performed by the sparse convolution processing in the processor. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of an information processing device according to a first embodiment of the present disclosure. [Figure 2] 4 is a flowchart showing the procedure of a power amount control process in the first embodiment. [Figure 3] FIG. 10 is a block diagram showing a schematic configuration of an information processing device according to a second embodiment of the present disclosure. [Figure 4] 10 is a flowchart showing the procedure of a power amount control process in the second embodiment. [Figure 5] 10 is a flowchart showing the procedure of a power amount control process in the third embodiment. [Figure 6] 10 is a flowchart showing the procedure of a power amount control process in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] A. First embodiment: A1.Device configuration: An information processing device 100 shown in FIG. 1 performs sparse convolution processing on image data acquired by an image sensor 10. The information processing device 100 is mounted on a moving object such as a vehicle, an automated guided vehicle, or a robot. The image sensor 10 is a sensor that outputs data that can be subjected to sparse convolution processing in a time series, such as an event camera, a LiDAR (Light Detection and Ranging), or an RGB-D camera. The image data that has undergone sparse convolution processing is used for any application, such as object recognition in autonomous driving or distance estimation to an object. The information processing device 100 includes an input / output interface (I / F) unit 1, an NPU (Neural Processing Unit) 2, a CPU (Central Processing Unit) 3, and a memory 4. The input / output I / F unit 1, the NPU 2, the CPU 3, and the memory 4 are all configured to be able to communicate with each other via a bus 7.

[0010] The input / output I / F unit 1 has a group of interfaces for electrically connecting to the image sensor 10. Image data acquired by the image sensor 10 is input to the information processing device 100 via the input / output I / F unit 1.

[0011] The NPU 2 is a processor that executes neural network processing. In the present disclosure, the NPU 2 executes sparse convolution processing on image data acquired in time series by the image sensor 10. Since the sparse convolution processing involves many multiply-accumulate operations, the NPU 2 is equipped with many (e.g., 256) multiply-accumulate operators.

[0012] The memory 4 is configured by, for example, a dynamic random access memory (DRAM) or a flash memory, and stores control programs for each unit of the information processing device 100. The CPU 3 reads and executes the control programs from the memory 4, thereby functioning as a size prediction unit 5 and a power consumption control unit 6.

[0013] The size prediction unit 5 predicts the rulebook size in sparse convolution processing. The "rulebook for sparse convolution processing" is a table linking elements of image data acquired by the image sensor 10, elements of kernels used in convolution, and elements of data output by the convolution. In sparse convolution processing, the NPU 2 performs a multiply-and-accumulate operation on image data acquired by the image sensor 10 according to a created rulebook. The "rulebook size" refers to the size of the rulebook. The rulebook size is determined by the size of the image data and the kernel used in convolution. In the present disclosure, the kernel is preset and the image data is acquired in chronological order, so the rulebook size can be predicted from the data rate of the image data. The "data rate" refers to the amount of data transferred per unit time. The data rate is expressed, for example, in bits per second (bps). The rulebook size can be predicted using, for example, a model that learns the relationship between the data rate of image data and the kernel, or a method such as simple regression analysis. In the following, the predicted rulebook size will also be referred to as the "predicted value."

[0014] The power amount control unit 6 uses the predicted value to output a control signal for controlling the amount of power supplied to the NPU 2. As described above, the NPU 2 performs product-sum operations according to the rulebook in sparse convolution processing. Because the processing load of the NPU 2 varies depending on the rulebook size, the power amount control unit 6 outputs a control signal for controlling the amount of power supplied to the NPU 2 according to the predicted rulebook size.

[0015] Specifically, the power amount control unit 6 compares the predicted value with a first upper threshold and a first lower threshold, and outputs a control signal according to the comparison result. The first upper threshold and the first lower threshold are preset thresholds. If the predicted value exceeds the first upper threshold, the power amount control unit 6 outputs a control signal to the NPU2 to increase the amount of power supplied to the NPU2. If the predicted value is less than the first lower threshold, the power amount control unit 6 outputs a control signal to the NPU2 to decrease the amount of power supplied to the NPU2. If the predicted value is within a range between the first upper threshold and the first lower threshold, the power amount control unit 6 does not output a control signal to control the amount of power supplied to the NPU2.

[0016] The control signal output by the power amount control unit 6 is 1. A control signal that controls the operating frequency of NPU2; 2. A control signal that controls the power gating of the arithmetic unit installed in NPU2; 3. A control signal that controls the power supply voltage of NPU2; Contains at least one of the following:

[0017] For example, when increasing the amount of power supplied to the NPU 2, the power amount control unit 6 outputs at least one of a control signal to increase the operating frequency, a control signal to cancel power gating, and a control signal to increase the power supply voltage to the NPU 2. On the other hand, when decreasing the amount of power supplied to the NPU 2, the power amount control unit 6 outputs at least one of a control signal to decrease the operating frequency, a control signal to execute power gating of a computing unit that does not need to operate, and a control signal to decrease the power supply voltage to the NPU 2.

[0018] Control of the operating frequency of the NPU2, control of power gating of the arithmetic unit mounted on the NPU2, and control of the power supply voltage of the NPU2 differ in the time required from the output of a control signal to actually affecting the amount of power. Specifically, control of the operating frequency of the NPU2 takes several nanoseconds. Control of power gating takes several microseconds. Control of the power supply voltage of the NPU2 takes hundreds of microseconds to several milliseconds. Therefore, one of the three methods described above is selected depending on the time required for sparse convolution of image data. The time required for sparse convolution may vary depending on the application using the sparsely convolved data. For example, when an application requiring a very short time for sparse convolution is executed, only control of the operating frequency of the NPU2, which requires the shortest control time, may be executed. Furthermore, when an application allowing a relatively long time for sparse convolution is executed, control of the operating frequency of the NPU2, control of power gating, and control of the power supply voltage of the NPU2 may be executed in parallel. The more these controls are performed, the more power savings can be expected.

[0019] A2. Power consumption control process: When the information processing device 100 acquires image data from the image sensor 10, the information processing device 100 executes the power consumption control process shown in FIG.

[0020] The size prediction unit 5 acquires the data rate of the image data output in time series by the image sensor 10 (step S105).

[0021] The size prediction unit 5 predicts the rulebook size in the sparse convolution process using the data rate (step S110). The rulebook size is predicted by a method such as the learning model or simple regression analysis described above.

[0022] The power amount control unit 6 compares the predicted value with a first upper threshold and a first lower threshold set in advance (step S115). If the predicted rulebook size is less than the first lower threshold set in advance, the power amount control unit 6 outputs a control signal to the NPU 2 to reduce the amount of power supplied to the NPU 2 (step S120). If the predicted rulebook size exceeds the first upper threshold set in advance, the power amount control unit 6 outputs a control signal to the NPU 2 to increase the amount of power supplied to the NPU 2 (step S125). If the predicted rulebook size is within the range between the first upper threshold and the first lower threshold set in advance, the power amount control unit 6 ends the process without controlling the amount of power.

[0023] According to the information processing device 100 of the first embodiment described above, prior to the sparse convolution processing of image data by the NPU2, the size prediction unit 5 predicts the rulebook size of the image data, and the power control unit 6 outputs a control signal to the NPU2 for controlling the amount of power supplied to the NPU2 according to the predicted rulebook size, thereby optimizing the amount of power supplied according to the processing load of the NPU2.

[0024] In addition, the signal for controlling the amount of power consumption includes at least one of a control signal for controlling the operating frequency of the NPU2, a control signal for controlling the power gating of the arithmetic unit mounted on the NPU2, and a control signal for controlling the power supply voltage of the NPU2.Therefore, by outputting one or more control signals selected according to the processing power or processing time required for the NPU2, the amount of power consumption can be controlled more appropriately.

[0025] B. Second embodiment: B1. Equipment configuration: An information processing device 100b of the second embodiment shown in FIG. 3 differs from the information processing device 100 of the first embodiment in that it further includes a maximum value prediction unit 8 that predicts the maximum value of the rulebook size, and in that a power amount control unit 6 outputs a control signal for controlling the amount of power supplied to the NPU 2 in accordance with the maximum value of the rulebook size predicted by the maximum value prediction unit 8. The image sensor 10 in the second embodiment is an event camera. The information processing device 100b and the event camera of the second embodiment are mounted on a vehicle. Other configurations of the information processing device 100b of the second embodiment are the same as those of the information processing device 100 of the first embodiment, so the same components are denoted by the same reference numerals and detailed description thereof will be omitted. The maximum value prediction unit 8 functions when the CPU 3 reads and executes a control program from the memory 4.

[0026] The maximum value prediction unit 8 predicts the maximum value of the rulebook size using environmental information about the event camera. In this disclosure, "environmental information about the event camera" refers to information that includes factors that affect the brightness change of the image data. Such information includes: 1. The time when the image data was acquired, 2. The location information of the event camera when the image data was acquired, 3. Weather information around the event camera when image data was acquired; 4. Steering information of vehicles equipped with event cameras; 5. Information on the number of objects within the angle of view of the event camera when the image data was acquired; Contains at least one of the following:

[0027] An event camera captures changes in brightness within its capture range and outputs image data in a time series. Brightness changes become larger, for example, when the event camera is in a dark environment and is affected by external lighting, or when the capture range of the event camera moves as the vehicle turns. As brightness changes become larger, the data rate of the image data also increases. This increases the rulebook size that NPU2 must process. Therefore, the maximum rulebook size can be predicted by using the environmental information of the event camera. Note that the predicted maximum rulebook value will be referred to below as the "predicted maximum value."

[0028] The "time when the image data is acquired" includes information indicating a time period such as morning, noon, evening, or night. The time when the image data is acquired may be acquired from a vehicle equipped with the information processing device 100b, or may be acquired via communication from an external server or the like. If the time when the image data is acquired is during bright hours such as daytime, the maximum value prediction unit 8 predicts that the maximum value of the rulebook size will be small because the luminance change of the image data will be small. On the other hand, if the time when the image data is acquired is during dark hours such as nighttime, large luminance changes are likely to occur in areas where artificial light sources such as LED lights are present due to the influence of the light source flickering, so the maximum value prediction unit 8 predicts that the maximum value of the rulebook size will be large.

[0029] The "location information of the event camera" includes information indicating whether the event camera is indoors or outdoors. The location information of the event camera may be acquired from a vehicle equipped with the information processing device 100b, or may be acquired via communication from an external server or the like. Furthermore, image data acquired by the event camera may be used to determine whether the event camera is indoors or outdoors by object recognition, such as a roof or tunnel. If the event camera is outdoors when the image data is acquired, the maximum value prediction unit 8 predicts that the maximum value of the rulebook size will be small because the luminance change in the image data will be small. On the other hand, if the event camera is indoors when the image data is acquired, the maximum value prediction unit 8 predicts that the maximum value of the rulebook size will be large because the luminance change in the image data will be large. Indoors, large luminance changes are likely to occur due to the presence of flashing light sources such as LED lights.

[0030] "Weather information around the event camera" includes information indicating whether the weather around the event camera is sunny, cloudy, rainy, snowy, etc. Weather information around the event camera may be acquired from a vehicle equipped with the information processing device 100b, or may be acquired via communication from an external server, etc. Weather information may also be acquired together with the location information described above. If the weather around the event camera when the image data is acquired is sunny, the maximum value prediction unit 8 predicts that the maximum value of the rulebook size will be small because the luminance change in the image data will be small. On the other hand, if the weather around the event camera when the image data is acquired is rainy, snowy, etc., the maximum value prediction unit 8 predicts that the luminance change in the image data will be large because the luminance change in the image data will be large. The reason why the luminance change in the image data is large when the weather around the event camera is rainy, snowy, etc. is because the rain, snow, etc. captured in the image is observed as an event.

[0031] "Vehicle steering information" is information including the steering angle of a vehicle equipped with an event camera. The vehicle steering information is acquired, for example, from a CAN (Controller Area Network) bus equipped in the vehicle. When the steering angle of the vehicle's steering is large, the shooting range of the event camera also moves significantly, resulting in a large change in brightness of the image data. As a result, the maximum value prediction unit 8 predicts that the maximum value of the rule book size will be large when the steering angle of the vehicle's steering is large. On the other hand, when the steering angle of the vehicle's steering is small, the movement of the shooting range of the event camera is small and the change in brightness of the image data is also small, resulting in a small prediction of the maximum value of the rule book size.

[0032] "Information on the number of objects within the angle of view of the event camera" includes information on the number of moving objects within the angle of view of the event camera. The number of moving objects within the angle of view of the event camera can be obtained using common methods such as object detection. When there are a large number of moving objects within the angle of view of the event camera, the maximum value prediction unit 8 predicts that the maximum value of the rulebook size will be large because the brightness change in the image data will be large. On the other hand, when there are a small number of moving objects within the angle of view of the event camera, the maximum value prediction unit 8 predicts that the maximum value of the rulebook size will be small because the brightness change in the image data will be small.

[0033] The maximum value prediction unit 8 may predict the maximum value of the rulebook size by using a model that has learned the relationship between each of the environmental information of the event camera described above and the maximum value of the rulebook size, or by using a method such as simple regression analysis.

[0034] The power amount control unit 6 uses the predicted maximum value to output a control signal for controlling the amount of power supplied to the NPU 2. Specifically, the power amount control unit 6 performs this control according to the result of comparing the predicted maximum value with a second upper threshold and a second lower threshold. The second upper threshold and the second lower threshold are preset thresholds. If the predicted maximum value exceeds the second upper threshold, the power amount control unit 6 outputs a control signal to the NPU 2 for increasing the amount of power supplied to the NPU 2. If the predicted maximum value is less than the second lower threshold, the power amount control unit 6 outputs a control signal to the NPU 2 for decreasing the amount of power supplied to the NPU 2. If the predicted maximum value is within the range between the second upper threshold and the second lower threshold, the power amount control unit 6 does not output a control signal for controlling the amount of power supplied to the NPU 2.

[0035] B2. Power control process: 4, the power amount control process of the second embodiment differs from the power amount control process of the first embodiment in that steps S205 to S225 are executed before step S105. Since the other steps in the power amount control process of the second embodiment are the same as those in the first embodiment, the same steps are denoted by the same reference numerals and detailed descriptions thereof will be omitted.

[0036] The maximum value prediction unit 8 acquires environmental information about the event camera (step S205).

[0037] The maximum value prediction unit 8 predicts the maximum value of the rulebook size in the sparse convolution process using the acquired environmental information (step S210).

[0038] The power amount control unit 6 compares the predicted maximum value with a second upper threshold and a second lower threshold that are set in advance (step S215). If the predicted maximum value is less than the second lower threshold that is set in advance, the power amount control unit 6 outputs a control signal to the NPU 2 to reduce the amount of power supplied to the NPU 2 (step S220). On the other hand, if the predicted maximum value exceeds the second upper threshold that is set in advance, the power amount control unit 6 outputs a control signal to the NPU 2 to increase the amount of power supplied to the NPU 2 (step S225). If the predicted maximum value is within the range between the second upper threshold and the second lower threshold that are set in advance, the power amount control unit 6 does not control the amount of power.

[0039] The information processing device 100b of the second embodiment described above has the same effects as the information processing device 100 of the first embodiment. Furthermore, according to the information processing device 100b of the second embodiment, the maximum value prediction unit 8 predicts the maximum value of the rulebook size using environmental information of the image sensor 10, and the power amount control unit 6 outputs a control signal for controlling the amount of power supplied to the NPU 2 according to the predicted maximum value, so that the amount of power can be appropriately controlled by two-stage control signals, namely, a control signal according to the predicted value and a control signal according to the predicted maximum value.

[0040] In addition, the environmental information used for maximum value prediction includes at least one of the following: the time when the image data was acquired, the location information of the event camera when the image data was acquired, the weather information around the event camera when the image data was acquired, the steering information of the vehicle in which the event camera is installed, and the information on the number of objects within the angle of view of the event camera when the image data was acquired.Therefore, by using appropriate information depending on the environment around the event camera, the accuracy of maximum value prediction can be further improved.

[0041] C. Third embodiment: 5, the power consumption control process in the information processing device 100 of the third embodiment differs from that of the second embodiment in that a comparison between the predicted value and the predicted maximum value is further performed in the process of step S115, and in that the information processing device 100 of the third embodiment further includes a process of step S130. The information processing device 100 of the third embodiment also differs from the information processing device 100b of the second embodiment in that it further includes a buffer memory. Since the other steps in the power consumption control process of the third embodiment are the same as those of the second embodiment, the same steps are assigned the same reference numerals and detailed descriptions thereof will be omitted.

[0042] In step S115, the power control unit 6 further compares the predicted value predicted in step S110 with the predicted maximum value predicted in step S210. If the predicted value exceeds the predicted maximum value, the power control unit 6 buffers the image data acquired by the image sensor 10 in a buffer memory and outputs a control signal to the NPU 2 to increase the amount of power supplied to the NPU 2 (step S130). Note that the comparison of the predicted value with the predicted maximum value in step S115 is performed before the comparison of the predicted value with the first and second thresholds. If the predicted value exceeds the predicted maximum value, the NPU 2 may not be able to complete the sparse convolution process on the image data due to insufficient power supplied to the NPU 2, resulting in insufficient processing capacity of the NPU 2. Therefore, in step S130, the image data is temporarily buffered in a buffer memory, and the power control unit 6 outputs a processing signal to the NPU 2 to increase the supplied power. Since the processing power of the NPU2 is improved by increasing the supplied power, the NPU2 can retrieve the buffered image data from the buffer memory and perform sparse convolution processing following step S130.

[0043] The information processing device 100 of the third embodiment described above achieves the same effects as the information processing device 100b of the second embodiment. Furthermore, according to the information processing device 100 of the third embodiment, even if the predicted value exceeds the maximum predicted value, i.e., if prediction of the maximum rulebook size fails, the image data is buffered in the buffer memory and the power amount control unit 6 outputs a processing signal to the NPU 2 to increase the supplied power, so that sparse convolution processing of the image data can be performed.

[0044] D. Fourth embodiment: 6, the power consumption control process in the information processing device 100 of the fourth embodiment differs from the power consumption control process of the third embodiment in that it includes a process of step S140 instead of the process of step S130. Since the other steps in the power consumption control process of the fourth embodiment are the same as those of the third embodiment, the same steps are denoted by the same reference numerals and detailed descriptions thereof will be omitted.

[0045] In step S115, the power control unit 6 further compares the predicted value predicted in step S110 with the maximum predicted value predicted in step S210. If the predicted value exceeds the maximum predicted value, the power control unit 6 discards the image data and outputs a control signal to the NPU 2 to increase the amount of power supplied to the NPU 2 (step S140). Note that the comparison of the predicted value with the maximum predicted value in step S115 precedes the comparison of the predicted value with the first and second thresholds. As described in the third embodiment, if the predicted value exceeds the maximum predicted value, the power supply to the NPU 2 may be insufficient, resulting in insufficient processing capacity for the NPU 2, which may prevent the NPU 2 from completing the sparse convolution process for the image data. In the power control process of the fourth embodiment, the image data for which the rulebook size has been predicted is discarded without being processed by the NPU 2, and the power control unit 6 outputs a processing signal to the NPU 2 to increase the supplied power. Next, the process returns to step S105, where the data rate of the new image data is acquired.

[0046] The information processing device 100 of the fourth embodiment described above provides the same effects as the information processing device 100b of the second embodiment. Furthermore, according to the information processing device 100 of the fourth embodiment, even if the predicted value exceeds the maximum predicted value, i.e., if prediction of the maximum rulebook size fails, the NPU 2 discards the image data without performing sparse convolution processing, and the size prediction unit 5 obtains the data rate of new image data and starts the processing again, so that the power consumption control process can be continuously executed.

[0047] E. Other Embodiments: (E1) In each embodiment, the first upper threshold and the first lower threshold may be determined by a general statistical method that targets the rulebook size and the amount of power corresponding to the processing capacity of the NPU 2. The second upper threshold and the second lower threshold may be determined by a general statistical method that targets the maximum value of the rulebook size and the amount of power corresponding to the processing capacity of the NPU 2.

[0048] (E2) In each embodiment, the control signal for controlling the amount of power may be a combination of two or more of the above-described control signals. By performing multiple power control operations, the amount of power supplied to the NPU 2 can be varied to a greater extent, allowing a more appropriate amount of power to be supplied to the NPU 2 depending on the processing load of the NPU 2.

[0049] (E3) In the second to fourth embodiments, the environmental information of the event camera may be a combination of two or more of the above-described environmental information. By using multiple pieces of environmental information according to the surrounding environment of the event camera, the accuracy of maximum value prediction can be further improved.

[0050] (E4) In the second to fourth embodiments, the information processing device 100b is configured to be mounted on a vehicle, but instead of this configuration, the information processing device 100b may be provided outside the vehicle. Such an information processing device 100b is, for example, a server installed outside the vehicle.

[0051] (E5) In steps S220 and S225 of the second to fourth embodiments, only the control signal for controlling the power supply voltage of the NPU 2 may be used as the control signal among the control signals for controlling the amount of power described above. As described above, power supply voltage control takes several hundred microseconds to several milliseconds from the time the control signal is input until the actual amount of power changes, and the time required for the amount of power to change is longer than that of the other two control signals. In the power amount control process of the second to fourth embodiments, power amount control in step S220 or step S225 is performed prior to power amount control according to the predicted value in steps S105 to S125. Therefore, even when power amount control is required in a shorter time, power amount control by controlling the power supply voltage, which takes a relatively long time, can be used.

[0052] (E6) In the second to fourth embodiments, the maximum value prediction unit 8 may determine whether to predict the maximum value of the rulebook size prior to the process of step S205. When the image data sparsely convolved by the NPU 2 is used for an application that requires shorter processing times, such as a vehicle collision avoidance assistance application, performing maximum value prediction before controlling the amount of power consumption may not satisfy the requirements of the application. Therefore, the maximum value prediction unit 8 determines whether to predict the maximum value depending on the application that uses the image data sparsely convolved by the NPU 2. If the maximum value prediction unit 8 determines that the application requires shorter processing times, it continues to execute the process of step S205. On the other hand, if the maximum value prediction unit 8 determines that the application does not require shorter processing times, it does not execute the processes of steps S205 to S215, and the size prediction unit 5 continues to execute the process of step S105. According to this embodiment, it is possible to determine whether to predict the maximum value of the rulebook size depending on the processing time required by the application that uses the image data that has been sparsely convolved.

[0053] (E7) In step S130 of the third embodiment, the image data is buffered in the buffer memory, but it may be buffered in the memory 4 in which various control programs are stored.

[0054] (E8) In step S130 of the third embodiment and step S140 of the fourth embodiment, the control signal for increasing the amount of power may be all of the control signals described above. When the process of step S130 is executed, that is, when the predicted value exceeds the predicted maximum value, the NPU 2 is required to have a large processing capacity. Therefore, by having the power amount control unit 6 output all of the control signals for increasing the amount of power supply, the processing capacity of the NPU 2 can be maximized in terms of the amount of power supply.

[0055] The size prediction unit 5, power amount control unit 6, maximum value prediction unit 8, and power amount control process described in the present disclosure may be implemented by a special-purpose computer configured by configuring a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the size prediction unit 5, power amount control unit 6, maximum value prediction unit 8, and power amount control process described in the present disclosure may be implemented by a special-purpose computer configured by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the size prediction unit 5, power amount control unit 6, maximum value prediction unit 8, and power amount control process described in the present disclosure may be implemented by one or more special-purpose computers configured by combining a processor and memory programmed to perform one or more functions with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory tangible recording medium.

[0056] The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, technical features in each embodiment corresponding to technical features in the embodiments described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. The present disclosure can be realized in the form of, for example, a power control method, a computer program for implementing such a method, a non-transitory recording medium on which such a computer program is recorded, etc. [Explanation of symbols]

[0057] 1...input / output I / F unit, 2...NPU, 3...CPU, 4...memory, 5...size prediction unit, 6...power amount control unit, 7...bus, 8...maximum value prediction unit, 10...image sensor, 100, 100b...information processing device

Claims

1. An information processing device (100, 100b) that performs sparse convolution processing on image data acquired in time series by an image sensor (10), a processor (2) that executes the sparse convolution processing operation; a size prediction unit (5) that predicts a rulebook size in the sparse convolution processing using a data rate of the image data acquired by the image sensor; an electric power amount control unit (6) that outputs a control signal to the processor for controlling the amount of electric power supplied to the processor according to a result of the rulebook size prediction by the size prediction unit; An information processing device comprising:

2. 2. The information processing device according to claim 1, A maximum value prediction unit (8) is further provided which predicts a maximum value of the rulebook size using environmental information around the image sensor, The power amount control unit outputs the control signal according to a result of the maximum value prediction by the maximum value prediction unit.

3. 3. The information processing device according to claim 2, The environmental information is The time when the image sensor acquires the image data; position information of the image sensor at the time of acquiring the image data; Weather information around the image sensor when the image data is acquired by the image sensor; and steering information of a vehicle equipped with the image sensor; Information on the number of objects within the angle of view of the image sensor when the image data of the image sensor is acquired; An information processing device that is any one of the above.

4. 3. The information processing device according to claim 2, The environmental information is The time when the image sensor acquires the image data; position information of the image sensor at the time of acquiring the image data; Weather information around the image sensor when the image data is acquired by the image sensor; and steering information of a vehicle equipped with the image sensor; Information on the number of objects within the angle of view of the image sensor when the image data of the image sensor is acquired; An information processing device including any two or more of the above.

5. The information processing device according to any one of claims 2 to 4, a buffer memory that buffers the image data acquired by the image sensor; If the rulebook size predicted by the size prediction unit exceeds the maximum value of the rulebook size predicted by the maximum value prediction unit, the buffer memory buffers the image data acquired by the image sensor; The power amount control unit outputs the control signal for increasing the amount of power supplied to the processor.

6. The information processing device according to any one of claims 2 to 4, If the rulebook size predicted by the size prediction unit exceeds the maximum value of the rulebook size predicted by the maximum value prediction unit, The processor discards the image data acquired by the image sensor without performing the sparse convolution process; The power amount control unit outputs the control signal for increasing the amount of power supplied to the processor.

7. The information processing device according to any one of claims 1 to 4, The control signal is a control signal for controlling the operating frequency of the processor; a control signal for controlling power gating; a control signal for controlling a power supply voltage of the processor; An information processing device including at least one of the above.

Images