Information processing device

Abstract

The information processing device of the present invention reduces the computational amount of object extraction performed in a DNN according to the driving environment, etc. in the vehicle's autonomous driving system. The present invention is an information processing device having a processor, a memory, and an arithmetic device that performs arithmetic operations using an inference model. The information processing device includes: a DNN processing unit that receives external information and extracts external objects from the external information through the inference model; and a processing content control unit that controls the processing content of the DNN processing unit. The DNN processing unit includes an object extraction unit that executes the inference model using a deep neural network having a layer with multiple neurons. The processing content control unit includes an execution layer determination unit that determines the layer used in the object extraction unit.

Description

Technical Field

[0001] The present invention relates to an information processing device using a deep neural network. Background Art

[0002] In recent years, technologies for controlling a vehicle to a destination by using machine learning through surrounding recognition, automatic steering, and automatic speed control have been advancing. In addition, as a method of machine learning applied to object recognition and the like, a deep neural network (DNN: Deep Neural Network) is known.

[0003] The DNN performs learning processing and inference processing. The learning processing is to grasp the features of an object, and the inference processing is to extract an object by using image data based on the result obtained by learning. Conventionally, when performing autonomous driving of a vehicle by using the DNN, first, image data of the outside world is acquired by using a camera and converted into a format that can be used in the DNN. In the inference processing, the converted image data is used as an input image, and the DNN that has completed the learning processing in advance is used to extract an object. Thereafter, a surrounding map is generated based on the object extraction result, and an action plan is formulated based on the surrounding map to control the vehicle.

[0004] Patent Document 1 discloses the following technique: extracting an object from an input image from a camera by using a neural network or the like and determining whether it is a drivable space. In addition, Patent Document 2 discloses the following technique: gradually reducing the number of pixels of an input image from a camera according to the driving state.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2019-008796

[0008] Patent Document 2: Japanese Patent Application Laid-Open No. 2018-056838 Summary of the Invention

[0009] Problems to be Solved by the Invention

[0010] Since the DNN repeatedly performs convolution operations consisting of multiplications and additions, the amount of computation is extremely large. In addition, especially in the autonomous driving of vehicles, the action plan needs to be continuously updated within an extremely short period of time. Therefore, object extraction under the DNN requires high-speed computation. As a result, when deploying the DNN in computing devices that can be mounted on vehicles, such as GPUs (Graphics Processing Unit), FPGAs (Field Programmable Gate Array), microcomputers, CPUs, etc., the power consumption and heat generation of the computing device are extremely large.

[0011] In addition, in order to continue autonomous driving in any scenario, object extraction with high precision is required. Therefore, the input image must use an image with a large number of pixels. When converting the input image, the power consumption and heat generation also increase. The power that a vehicle can provide is limited. Therefore, the increase in power consumption and heat generation will lead to a shortened driving range of electric vehicles. In addition, there will be a problem of increased cooling costs for the computing device.

[0012] However, the above-mentioned Patent Document 1 does not consider reducing power consumption in the conversion of the input image and object extraction. In addition, Patent Document 2 changes the conversion method of the input image according to the driving environment, thereby reducing the power in the image conversion process, but does not consider the power consumption in the inference process using the DNN with a particularly large amount of computation. Therefore, a sufficient power consumption reduction effect cannot be expected.

[0013] In view of the above problems, an object of the present invention is to determine whether the object extraction process under the DNN can be omitted according to the operation load state of the computing device mounted in the vehicle and the driving environment, thereby reducing the operation load and power consumption of the computing device and suppressing heat generation.

[0014] Technical means for solving the problem

[0015] The present invention is an information processing device having a processor, a memory, and a computing device that performs operations using an inference model. The information processing device includes: a DNN processing unit that receives external information and extracts external objects from the external information through the inference model; and a processing content control unit that controls the processing content of the DNN processing unit. The DNN processing unit has an object extraction unit that executes the inference model using a deep neural network with layers having multiple neurons. The processing content control unit includes an execution layer determination unit that determines the layers used in the object extraction unit.

[0016] Effects of the invention

[0017] The information processing device of the present invention reduces the amount of computation of a deep neural network (DNN) according to the driving environment. Thus, while ensuring the object extraction accuracy of the DNN required for autonomous driving, it is possible to reduce the power consumption of the computing device and suppress the heat generation. As a result, it is possible to increase the driving range of an electric vehicle and reduce the cost of the cooling system.

[0018] Details of at least one implementation of the subject matter disclosed in this specification are described in the accompanying drawings and the following description. Other features, aspects, and effects of the disclosed subject matter will become apparent from the following disclosure, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 FIG. 1 shows Example 1 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0020] Figure 2 FIG. 2 shows Example 1 of the present invention and is a diagram showing a part of the process of object extraction using a DNN.

[0021] Figure 3 FIG. 3 shows Example 2 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0022] Figure 4 FIG. 4 shows Example 3 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0023] Figure 5 FIG. 5 shows Example 4 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0024] Figure 6 FIG. 6 shows Example 5 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0025] Figure 7 FIG. 7 shows Example 6 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0026] Figure 8 FIG. 8 shows Example 7 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0027] Figure 9 FIG. 9 shows Example 7 of the present invention and is a diagram showing a part of the process of object extraction.

[0028] Figure 10 FIG. 10 shows Example 8 of the present invention and is a block diagram showing an example of the configuration of an autonomous driving system.

[0029] Figure 11 FIG. 11 shows Example 8 of the present invention and is a diagram showing an image from a camera during tunnel driving.

[0030] Figure 12 Embodiment 9 of the present invention is a block diagram showing an example of the configuration of an autonomous driving system.

[0031] Figure 13 Embodiment 10 of the present invention is a block diagram showing an example of the configuration of an autonomous driving system.

[0032] Figure 14 Embodiment 11 of the present invention is a block diagram showing an example of the configuration of an autonomous driving system.

[0033] Figure 15 Embodiment 12 of the present invention is a block diagram showing an example of the configuration of an autonomous driving system. Detailed implementation manners

[0034] Hereinafter, embodiments will be described with reference to the drawings.

[0035] Embodiment 1

[0036] A deep neural network (hereinafter referred to as DNN: Deep Neural Network) having a plurality of neuron layers repeatedly performs a convolution operation composed of multiplication and addition, so the amount of computation is extremely large. Along with this, the power consumption and heat generation of the computing device that executes the DNN are large. Therefore, when performing object extraction processing using the DNN in the information processing device of the vehicle, problems such as a shortened driving distance of the electric vehicle and an increased cooling cost arise.

[0037] Figure 1 It is a block diagram showing an example of the configuration of an autonomous driving system using the information processing device 100 of the present embodiment.

[0038] As Figure 1 shown, the autonomous driving system using the information processing device 100 of the present embodiment is composed of an information processing device 100, a camera 200, and a vehicle control unit 300. Here, the information processing device 100 is a computer including a processor 10, a memory 20, and an FPGA (Field Programmable Gate Array) 170.

[0039] Furthermore, in this embodiment, an example of using the FPGA 170 as the computing device that executes the DNN is shown, but it is not limited thereto, and a GPU (Graphics Processing Unit) may also be used for the computing device. In addition, the computing device that executes the DNN may be housed in the same package as the processor 10. In addition, the processor 10 and the FPGA 170 are connected to the camera 200 and the vehicle control unit 300 via an interface (not shown).

[0040] The action plan unit 130 is loaded in the memory 20 in the form of a program for execution by the processor 10. The processor 10 executes processing according to the programs of the respective functional units, and thus operates as a functional unit that provides a prescribed function. For example, the processor 10 executes processing according to the action plan program, and thus functions as the action plan unit 130. The same applies to other programs. Furthermore, the processor 10 also operates as a functional unit that provides the functions of the respective processes executed by the respective programs. The computer and the computer system are devices and systems that include these functional units.

[0041] The FPGA 170 includes the DNN processing unit 110 and the processing content control unit 120. In addition, the DNN processing unit 110 includes the object extraction unit 111. The processing content control unit 120 includes the execution layer determination unit 121. Furthermore, the camera 200 may or may not be included in the information processing device 100. In addition, the information processing device 100 may or may not be mounted on a vehicle.

[0042] First, the processing of the DNN processing unit 110 will be described. The object extraction unit 111 holds a DNN (inference model) for inference processing that has been subjected to learning processing in a computer such as a PC or a server. The FPGA 170 extracts an object from the image data using the DNN that reflects the information output from the processing content control unit 120 described later, for the external information (image data) acquired by the camera 200.

[0043] Furthermore, the camera 200 is installed at a prescribed position (not shown) of the vehicle, acquires an image in front of the vehicle as external information, and outputs the image to the information processing device 100 to detect an object outside the vehicle (in front).

[0044] The action plan unit 130 generates an action plan such as the traveling direction and traveling speed of the vehicle using the information of the object extracted by the object extraction unit 111, and outputs the action plan to the vehicle control unit 300. The vehicle control unit 300 controls the vehicle according to the output from the action plan unit 130. Regarding the processing performed in the vehicle control unit 300, well-known or publicly known techniques can be applied, and thus will not be described in detail in this embodiment.

[0045] Next, the processing of the processing content control unit 120 will be described. The execution layer determination unit 121 holds the number of execution layers of the DNN used in the object extraction processing, and outputs this information to the object extraction unit 111.

[0046] Next, the processing of the object extraction unit 111 will be described using a specific example. In the object extraction processing of the DNN, objects are extracted using multiple layers with different numbers of segments for the external information (image data), whereby objects of various sizes can be extracted. Figure 2 To represent Figure 1A diagram showing an example of part of the process when the object extraction unit 111 extracts objects from the information (image data) of the camera 200 using DNN.

[0047] Figure 2 In [description], the camera image 500 represents an example of an image output from the Figure 1 camera 200 shown. In addition, the image 510 represents an example of segmenting the camera image 500 in the first layer of the DNN of the object extraction unit 111, and the images 520, 530, 540, and 550 represent the segmentation methods in the second, third, fourth, and fifth layers, respectively.

[0048] Figure 2 It shows the following: For the camera image 500, the DNN processing unit 110 extracts small objects 511 such as people and cars in the distance by segmenting the camera image smaller in the first layer, and large objects 551 such as cars in the vicinity by segmenting the camera image larger in the fifth layer.

[0049] The DNN processing unit 110 divides the input image (500) into 5×5 = 25 blocks in the image 510, and the object extraction unit 111 performs object extraction processing in units of these blocks. Similarly, the DNN processing unit 110 divides into 4×4 = 16 in the image 520, 3×3 = 9 in the image 530, 2×2 = 4 in the image 540, and 1×1 = 1 block in the image 550 to perform object extraction processing.

[0050] Therefore, the number of times the DNN processing unit 110 performs object extraction processing in the Figure 2 shown image is 25 + 16 + 9 + 4 + 1 = 55 times. Furthermore, the structure of the DNN only needs to be able to extract objects, and is not limited to a structure that first extracts small objects and then extracts large objects as in the structure used in this embodiment. In addition, regarding the number of block divisions and the number of layers during the operation in the DNN, as long as the accuracy required for autonomous driving can be ensured, it is not limited to the Figure 2 method.

[0051] On the other hand, for example, in the case of a vehicle traveling at a sufficiently low speed compared to ordinary roads on a road with good visibility within a private area such as a factory and with a limited variety of vehicles in motion, small objects in the distance several hundred meters ahead may not be extracted. Therefore, the number of blocks for object extraction processing can be reduced by reducing the number of layers of the DNN.

[0052] For example, the execution layer determination unit 121 determines the layers of the DNN executed in the object extraction process as the second to fifth layers excluding the first layer that extracts the smallest object and instructs the object extraction unit 111. In the object extraction unit 111, the object extraction process is performed by only executing the second to fifth layers of the DNN according to the information from the execution layer determination unit 121. In this case, the number of times the object extraction process is performed is 16 + 9 + 4 + 1 = 30, and 25 operations can be reduced compared to the case where the object extraction is performed in all the layers.

[0053] Furthermore, the execution layer determination unit 121 only needs to indicate a preset layer as a processing condition to the object extraction unit 111 based on the driving environment or the like at the start of driving.

[0054] In this way, by reducing the layers (fixed values) of the DNN used by the object extraction unit 111, it is possible to reduce the power consumption and suppress the heat generation of arithmetic devices such as the FPGA 170 or GPU, for example. As a result, the travelable distance of the electric vehicle increases, and the cooling cost of the arithmetic device can be reduced.

[0055] Furthermore, this time, for the sake of simplicity of explanation, the values of the number of layers and blocks of the DNN in Figure 2 are used as described above, but in the DNN used in actual object extraction, the external information is divided into thousands to tens of thousands of blocks, and the number of layers is also large. Therefore, the actual number of operations is much larger than the example shown above, and the effect of reducing the amount of operations brought about by reducing the number of layers is also extremely large.

[0056] Furthermore, in this embodiment, an example in which the camera 200 is used as the sensor for acquiring external information is shown, but any sensor such as LiDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), or far-infrared camera that can acquire the distance to an object and the type of the object can be used, and it is not limited to the camera 200. In addition, the sensors can be used alone or in combination.

[0057] In the above-described Embodiment 1, an example of the object extraction unit 111 that executes the inference model generated by machine learning in the DNN is shown, but it is not limited thereto, and a model generated by other machine learning or the like can be used.

[0058] Embodiment 2

[0059] Next, Embodiment 2 of the present invention will be described. Figure 3 A block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of Embodiment 2. Figure 3 In Figure 1The same constituent elements are given the same names and symbols, and unless otherwise specified, they are regarded as having the same or similar functions and the description is omitted.

[0060] This embodiment shows the following example: a processing condition determination unit 122 and a sensor for detecting external state information are added to the said Embodiment 1, and the execution layer determination unit 121 in the FPGA 170 is moved into the memory 20. Other configurations are the same as those in Embodiment 1.

[0061] Figure 3 In this, since the processing condition determination unit 122 newly added from Embodiment 1 uses information indicating the driving state from sensors of a running vehicle, etc. as input to determine the current driving environment. The execution layer determination unit 121 dynamically changes the number of layers (processing conditions) of the DNN used in the object extraction process according to the driving environment determined by the processing condition determination unit 122.

[0062] This embodiment shows an example in which a vehicle speed sensor 30 and an illuminance sensor 40 are connected to the information processing device 100 as sensors for detecting the driving state of the vehicle, but is not limited to these. As long as the vehicle speed can be detected at least, the vehicle speed can also be calculated based on position information such as GPS (Global Positioning System).

[0063] The processing condition determination unit 122 of the information processing device 100 determines the driving environment based on information from the vehicle speed sensor 30 and the illuminance sensor 40 and outputs it to the execution layer determination unit 121. The execution layer determination unit 121 determines the layers used in the DNN in the form of processing conditions according to the input driving environment and instructs the object extraction unit 111.

[0064] The example shown in Embodiment 1 is that since the number of layers (processing conditions) is a fixed value, the execution layer determination unit 121 is configured in the FPGA 170. In Embodiment 2, the execution layer determination unit 121 dynamically changes the number of layers (processing conditions). Therefore, the execution layer determination unit 121 is loaded into the memory 20 in the form of a program for the processor 10 to execute.

[0065] The example shown in Embodiment 1 is a case of statically determining the execution layer of the DNN in a restricted driving environment. However, in the autonomous driving of a vehicle, the driving environment is diverse and complex, and the accuracy of the objects extracted from the image data is also changing all the time. Therefore, it is more ideal to be able to dynamically change the execution layer of the DNN according to the driving environment.

[0066] Next, the processing of the processing condition determination unit 122 will be described using a specific example.

[0067] As an example, an example of using the vehicle speed detected by the vehicle speed sensor 30 as information indicating the driving state will be shown. The following example will be described: If the vehicle speed detected by the vehicle speed sensor 30 is equal to or lower than a preset vehicle speed threshold, the processing condition determination unit 122 determines that the current driving environment is congested.

[0068] When the vehicle is driving at an extremely low speed (for example, 5 km / h) in congestion, the change in the image input from the camera 200 is small. Therefore, the object extraction unit 111 only needs to recognize objects dozens of meters ahead, and does not need to perform object extraction with the accuracy for recognizing objects hundreds of meters ahead.

[0069] The execution layer decision unit 121 determines the processing conditions according to the driving environment determined by the processing condition determination unit 122. As the processing conditions, the execution layer decision unit 121 determines the layers of the DNN executed in the object extraction unit 111 in such a way that while ensuring the accuracy of object extraction required for autonomous driving, the amount of computation of the DNN is reduced. The layers are determined to be the second layer (520) to the fifth layer (550) excluding the first layer ( Figure 2 of the image 510) that detects the smallest object.

[0070] In this way, in the second embodiment, a sensor for detecting the driving state and a processing condition determination unit 122 for determining the driving environment of the vehicle according to the information indicating the driving state are added to the configuration of the first embodiment, and the execution layer decision unit 121 determines the processing conditions suitable for the driving environment. Thus, compared with the first embodiment, the information processing apparatus 100 of the second embodiment can dynamically change the accuracy of object extraction and the processing conditions that affect the power consumption and heat generation of the arithmetic device (FPGA 170) according to the driving environment.

[0071] As a result, compared with the first embodiment, the travelable distance of the electric vehicle increases, and a reduction in the cooling cost of the arithmetic device (FPGA 170) can be expected. Furthermore, the method of determining congestion above is not limited to the vehicle speed sensor 30, and may also be congestion information such as image information, radio, and Internet that knows the driving speed.

[0072] In addition, using the illuminance sensor 40 as the information indicating the driving state, if the surrounding brightness (illuminance) is equal to or lower than a preset illuminance threshold, the processing condition determination unit 122 determines that it is night. At night, the surroundings become dark, and compared with the case where the surroundings are bright, the camera 200 cannot obtain detailed external information. Therefore, external information capable of extracting objects hundreds of meters ahead cannot be obtained, so the object extraction unit 111 only needs to perform arithmetic operations with the accuracy of extracting objects dozens of meters ahead, and does not need to perform object extraction with the accuracy for recognizing objects hundreds of meters ahead.

[0073] In this case, the execution layer decision unit 121 determines the information of the layers of the DNN executed in the object extraction unit 111 to be the second layer (520) to the fifth layer (550) according to the driving environment determined by the processing condition determination unit 122. In addition, the information indicating the driving state for determining the current ambient brightness is not limited to the illuminance sensor 40. For example, sensors such as the camera 200 that can detect the surrounding brightness or means such as radio, Internet, and GPS that can determine the time can also be used to detect whether it is day or night.

[0074] In addition, an in-vehicle camera (not shown) can also be used as the information indicating the driving state. In this case, the processing condition determination unit 122 detects whether the driver is operating the steering wheel based on the image data of the in-vehicle camera. If the driver is operating the steering wheel, it is determined that autonomous driving (AD) is not being used. When using AD, the vehicle is controlled only by the autonomous driving system, so object extraction with high precision such as small objects needs to be recognized. In the case where AD is not used, the driver can know the external information, so it is sufficient to maintain the accuracy of object extraction that can achieve advanced driver assistance systems (ADAS).

[0075] The execution layer decision unit 121 determines the information of the layers of the DNN executed in the object extraction unit 111 to be the second layer (520) to the fifth layer (550) according to the driving environment determined by the processing condition determination unit 122.

[0076] In addition, the information for the processing condition determination unit 122 to determine whether AD is being used is not limited to the in-vehicle camera, and can also be a device such as a sensor that detects the operation of the steering wheel and can determine whether the driver is in the driver's seat. In addition, it can also be determined whether AD is being used based on data such as the presence or absence of the action plan of the autonomous driving vehicle generated by the action plan unit 130.

[0077] Furthermore, the processing condition determination unit 122 can use any one of the information indicating the driving state to determine the processing conditions, or can also use multiple pieces of information to determine the processing conditions.

[0078] Embodiment 3

[0079] Next, Embodiment 3 of the present invention will be described. Figure 4 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of the present embodiment. Figure 4 Among them, the same constituent elements as Figure 3 are given the same names and symbols, and unless otherwise specified, they are considered to have the same or similar functions and the description is omitted.

[0080] This embodiment shows an example in which a processing condition determination unit 122 and a load detection device 50 are added to the said Embodiment 1, and the execution layer determination unit 121 in the FPGA 170 is moved into the memory 20.

[0081] Other configurations are the same as those in Embodiment 1.

[0082] In the aforementioned Embodiment 2, the processing condition determination unit 122 determines the current driving environment by taking information indicating the driving state from sensors during driving as input. In this embodiment, the processing condition determination unit 122 determines the load state of the current FPGA 170 by taking the operation load of the object extraction unit 111 in the information processing device 100 as input. The execution layer determination unit 121 determines the number of layers (processing conditions) of the DNN used in the object extraction unit 111 according to the load state determined by the processing condition determination unit 122.

[0083] Figure 4 In this, the newly added load detection device 50 in Embodiment 1 detects the operation load of the FPGA 170 and inputs it to the processing condition determination unit 122. Furthermore, the load detection device 50 may include, for example, sensors that detect the power consumption, supply current, etc. of the FPGA 170 as the operation load.

[0084] In addition, the example shown in Embodiment 1 is that since the number of layers (processing conditions) is a fixed value, the execution layer determination unit 121 is configured in the FPGA 170. In Embodiment 3, the execution layer determination unit 121 dynamically changes the number of layers (processing conditions), and thus is loaded into the memory 20 in the form of a program for the processor 10 to execute.

[0085] Next, a specific example is used to illustrate the processing of the processing condition determination unit 122.

[0086] As an example, the processing condition determination unit 122 calculates the load rate of the FPGA 170 as the operation load and compares it with a specified load rate threshold. Furthermore, regarding the load rate, an example is shown in which the value (%) obtained by dividing the power consumption detected by the load detection device 50 by the specified maximum power is used.

[0087] When the load rate of the FPGA 170 is higher than the load rate threshold, the processing condition determination unit 122 determines to reduce the operation load of the FPGA 170. The execution layer determination unit 121 determines the information of the layers of the DNN used in the object extraction unit 111 in such a way that the load rate of the FPGA 170 becomes below the load rate threshold according to the determination result from the processing condition determination unit 122, which is the second layer (520) to the fifth layer (550) excluding the first layer (510) that detects the smallest object.

[0088] Thus, by adding a processing condition determination unit 122 that determines the load state based on the operation load of the FPGA 170, the load state corresponding to the load rate of the FPGA 170 included in the information processing apparatus 100 can be determined, and the execution layer determination unit 121 can dynamically change the processing conditions according to the load state. Accordingly, compared with the first embodiment, the information processing apparatus 100 can dynamically reduce the power consumption and suppress the heat generation of the FPGA 170.

[0089] Furthermore, in the information processing apparatus 100 of the present embodiment, as described above, it is possible to suppress the occurrence of an abnormality in the FPGA 170 due to thermal runaway or the like and prevent the reduction of the operation accuracy of the arithmetic unit. Accordingly, compared with the first embodiment, it is possible to increase the travel distance of the electric vehicle and reduce the cooling cost of the FPGA 170 or the like.

[0090] In addition, in the present embodiment, the load rate of the FPGA 170 as the operation load is not limited to power consumption. A value associated with the operation load such as the temperature of the FPGA 170 may be used. Further, the above operation load may include not only the operation load of the FPGA 170 but also the operation load of the processor 10.

[0091] Furthermore, the processing condition determination unit 122 may determine the load state using any one of the operation loads in the operation load, or may determine the load state using a plurality of operation loads.

[0092] Embodiment 4

[0093] Next, Embodiment 4 of the present invention will be described. Figure 5 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of the present embodiment. Figure 5 Among them, the same reference numerals and symbols are given to the same components as those in Figure 3 , Figure 4 and, unless otherwise specified, they are regarded as having the same or similar functions and the description thereof is omitted.

[0094] In the present embodiment, an example in which the configuration of the third embodiment is added to the configuration of the second embodiment is shown. Other configurations are the same as those in the second embodiment.

[0095] In the second embodiment, an example in which the current driving environment is determined by using information indicating the driving state from sensors or the like during driving as an input is shown, and in the third embodiment, an example in which the current load state is determined by using the operation load of the information processing apparatus 100 as an input is shown. In the present embodiment, an example in which the second embodiment is combined with the third embodiment is shown.

[0096] The processing condition determination unit 122 determines the driving environment and the load state by taking as inputs both the information indicating the driving state and the operation load of the FPGA 170, and outputs the determination result to the execution layer decision unit 121. The execution layer decision unit 121 determines the number of layers (processing conditions) of the DNN used in the object extraction unit 111 based on the determination result from the processing condition determination unit 122.

[0097] In this way, by determining the driving environment and the load state in the processing condition determination unit 122 based on the information indicating the driving state and the operation load of the FPGA 170, compared with Embodiment 2 and Embodiment 3, it is possible to dynamically change the object extraction accuracy and reduce the power consumption and heat generation.

[0098] As a result, it is possible to expect an effect that combines the advantages of Embodiment 2 and Embodiment 3. That is, it is possible to keep the load of the FPGA 170 in a normal state while maintaining an appropriate object extraction accuracy in various driving environments.

[0099] Furthermore, the processing condition determination unit 122 can determine the processing conditions using either the information indicating the driving state or the operation load, or can also determine the processing conditions using multiple ones.

[0100] Embodiment 5

[0101] Next, the configuration of Embodiment 5 of the present invention will be described. Figure 6 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of the present embodiment. Figure 6 Among them, the same reference numerals and symbols are given to the same components as Figure 5 those in, and unless otherwise specified, they are considered to have the same or similar functions and the description thereof is omitted.

[0102] This embodiment is obtained by adding a LiDAR 400, a sensor recognition unit 140, and a recognition accuracy determination unit 150 to the components of the above-described Embodiment 4. Other configurations are the same as those of Embodiment 4.

[0103] Figure 6 Among them, the newly added sensor recognition unit 140 in Embodiment 4 processes the information (distance and azimuth) from the LiDAR 400 to recognize an object. In addition, the recognition accuracy determination unit 150 determines the difference between the object extraction result from the object extraction unit 111 and the object recognition result from the sensor recognition unit 140, thereby determining whether the object extraction by the object extraction unit 111 maintains the required accuracy, and outputs the determination result to the processing condition determination unit 122.

[0104] Subsequently, the processing condition determination unit 122 updates the reference value that is the condition for reducing the computational load in the object extraction unit 111 based on the determination result from the recognition accuracy determination unit 150, the driving state, and the computational load, thereby preventing excessive omission of computations such as insufficient accuracy of object extraction caused by reduction of the computational load.

[0105] In the information processing apparatus 100 of Embodiments 1, 2, 3, and 4, the reference value for controlling the number of layers of the DNN used in the object extraction unit 111 based on the information indicating the driving state and the computational load is static. To maintain the accuracy of object extraction required for autonomous driving, it is desirable for this reference value to be dynamic.

[0106] Next, the processing of the recognition accuracy determination unit 150 will be described using a specific example.

[0107] As an example, when image data from the camera 200 is input, the number of objects extracted by the object extraction unit 111 is 5, and the number of objects recognized by the sensor recognition unit 140 based on the information from the LiDAR 400 is 7. The number of objects extracted by the object extraction unit 111 is less than the number of objects recognized by the sensor recognition unit 140.

[0108] In this case, since the objects recognized by the sensor recognition unit 140 are not extracted by the object extraction unit 111, the recognition accuracy determination unit 150 determines that the object extraction unit 111 does not maintain the accuracy of object extraction required for autonomous driving.

[0109] The recognition accuracy determination unit 150 outputs a command to change the processing conditions in a manner that improves the accuracy of object extraction to the processing condition determination unit 122 based on the information from the object extraction unit 111 and the sensor recognition unit 140, and the processing condition determination unit 122 changes the reference value for determining the processing conditions.

[0110] Furthermore, the recognition accuracy determination unit 150 may also calculate the value obtained by dividing the number of objects extracted by the object extraction unit 111 by the number of objects recognized by the sensor recognition unit 140 as the recognition accuracy. When the calculated recognition accuracy is below a preset accuracy threshold, it is determined that the recognition accuracy has decreased and is output to the processing condition determination unit 122.

[0111] Regarding the reference value of this embodiment, for example, when it is set as a value for correcting the load rate threshold, when the processing condition determination unit 122 receives a command to improve the accuracy of object extraction from the recognition accuracy determination unit 150, the reference value is set to +5% to increase the load rate threshold. In addition, the processing condition determination unit 122 notifies the execution layer determination unit 121 of the fact that the load rate threshold has been increased.

[0112] When the execution layer decision unit 121 receives a notification of an increase in the load rate threshold, it adds a specified value to the number of layers of the DNN currently being used in the object extraction unit 111 and instructs the object extraction unit 111 of the FPGA 170. The specified value for the execution layer decision unit 121 is set to, for example, 1 layer or the like.

[0113] In this way, the execution layer decision unit 121 can increase the number of layers of the DNN used in the object extraction unit 111 according to a request from the recognition accuracy determination unit 150, and improve the object extraction accuracy.

[0114] However, when the driving environment is congested or at night, the effect of increasing the number of layers of the DNN used in the object extraction unit 111 is low, so the execution layer decision unit 121 can cancel the notification of the increase in the load rate threshold.

[0115] In this way, by additionally using the sensor recognition unit 140 that identifies objects by using information from the LiDAR 400 and the recognition accuracy determination unit 150 that determines the object extraction accuracy of the object extraction unit 111 and the sensor recognition unit 140, it is possible to prevent excessive reduction of the arithmetic processing (layers) with insufficient accuracy of object extraction required for autonomous driving in the object extraction unit 111.

[0116] As a result, the information processing device 100 can maintain processing with higher accuracy than in the first, second, third, and fourth embodiments. As a result, the information processing device 100 of the present embodiment can further reduce power consumption compared to the first, second, third, and fourth embodiments.

[0117] Furthermore, in this embodiment, an example is shown in which the sensor used in the determination of the object extraction accuracy is the LiDAR 400, but as long as it is a sensor such as a camera, RADAR, far-infrared camera, etc. that can discriminate the distance to an object and the type of the object and is different from the sensor that outputs external information input to the object extraction unit 111, it is not limited to the LiDAR. In addition, the sensors can be used alone or in combination.

[0118] In addition, in this embodiment, as a method for the recognition accuracy determination unit 150 to determine the object extraction accuracy, it is a static process of determining whether the result of the object extraction unit 111 is consistent with the result of the sensor recognition unit 140, but the determination criterion for the object extraction accuracy can also be dynamically changed by making the recognition accuracy determination unit 150 have a learning function.

[0119] Embodiment 6

[0120] Next, Embodiment 6 of the present invention will be described. Figure 7 It is a block diagram showing an example of the configuration of an autonomous driving system using the information processing device 100 of this embodiment. Figure 7 In, forFigure 5 The same constituent elements are given the same names and symbols, and are regarded as having the same or similar functions and the description is omitted as long as there is no special explanation.

[0121] In this embodiment, a description is given of Figure 5 an example in which an additional information determination unit 123 is added to the processing content control unit 120 shown in the above-described Embodiment 4, and an external information conversion unit 112 is added to the DNN processing unit 110, and the number of pixels (or resolution) of the external image input from the camera 200 is changed according to the driving environment and the load state. Other configurations are the same as those in Embodiment 4.

[0122] Figure 7 Herein, the additional information determination unit 123 and the external information conversion unit 112 newly added from Embodiment 4 are described. The additional information determination unit 123 determines the number of pixels (resolution) of the external image acquired by the camera 200 to be input to the object extraction unit 111 based on the determination result from the processing condition determination unit 122, and issues an instruction to the external information conversion unit 112 of the FPGA 170.

[0123] The external information conversion unit 112 converts the external image acquired by the camera 200 into a format usable in the DNN according to the instruction from the additional information determination unit 123. Thereby, object extraction is performed in the object extraction unit 111 with the number of pixels corresponding to the processing condition (number of layers).

[0124] In Embodiments 1, 2, 3, and 4 described above, the sizes of the objects extracted in each layer (510 to 550) of the object extraction unit 111 are different. Therefore, depending on the processing content of each layer of the object extraction unit 111, the required number of pixels (resolution) of the external image is also different. Therefore, it is desirable that the number of pixels of the image input to the object extraction unit 111 be changed dynamically according to the number of layers used in the object extraction process.

[0125] Next, the processing in the external information conversion unit 112 will be described using a specific example.

[0126] As an example, the camera 200 outputs an image of 2 million pixels. In addition, the additional information determination unit 123 determines that the number of pixels required for object extraction is 250,000 pixels based on the determination result from the processing condition determination unit 122.

[0127] In this case, the external information conversion unit 112 converts the 2 million pixel image output from the camera 200 to 250,000 pixels. The object extraction unit 111 uses the 250,000 pixel image output from the external information conversion unit 112 to perform object extraction.

[0128] Thus, the DNN processing unit 110 dynamically changes the number of pixels of the input image sent to the object extraction unit 111 according to the determination result from the processing condition determination unit 122, so that an image with the number of pixels corresponding to the number of execution layers of the DNN used in the object extraction unit 111 can be used. Therefore, in addition to reducing the number of execution layers of the DNN, the number of pixels of the image in the object extraction unit 111 can also be reduced, thereby reducing the amount of computation.

[0129] Accordingly, the information processing apparatus 100 of the present embodiment can achieve reduction of power consumption and suppression of heat generation as compared with the first, second, third, and fourth embodiments. As a result, as compared with the first, second, third, and fourth embodiments, the information processing apparatus 100 of the present embodiment increases the travelable distance of the electric vehicle, and a reduction in the cooling cost of the FPGA 170 or the like can be expected.

[0130] Furthermore, in the present embodiment, the information amount determination unit 123 changes the number of pixels from the camera 200, and if the external information output is LiDAR or RADAR, a change (reduction) in the information amount of the external information such as a change in the number of point clouds is sufficient, and it is not limited to the number of pixels.

[0131] Embodiment 7

[0132] Figure 8 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of Embodiment 7. Figure 8 In, the same Figure 1 constituent elements are given the same names and symbols, and unless otherwise specified, they are considered to have the same or similar functions and the description thereof is omitted.

[0133] In the present embodiment, an example in which the execution layer determination unit 121 of the first embodiment is replaced with a block region determination unit 124 is shown. Other configurations are the same as those of the first embodiment.

[0134] Figure 8 The newly added block region determination unit 124 in stores a block region for performing object extraction processing of the DNN, and outputs the information of the block region to the object extraction unit 111. The block region determination unit 124 of the present embodiment stores a preset region in the form of a fixed block region.

[0135] Next, the processing of the object extraction unit 111 will be described using a specific example.

[0136] Figure 9 To show Figure 8 FIG. is an example showing a part of the processing of the object extraction unit 111 that extracts an object from the image data input from the camera 200 by means of the DNN.

[0137] Figure 9In this case, the camera image 600 represents the camera image output from the camera 200 in Figure 8 . Further, the image 610 represents the segmentation state of the camera image 600 in the first layer of the DNN used by the object extraction unit 111, the image 620 represents the segmentation state in the second layer, the image 630 represents the segmentation state in the third layer, the image 640 represents the segmentation state in the fourth layer, and the image 650 represents the segmentation state in the fifth layer.

[0138] The DNN processing unit 110 divides the camera image 600 into 5×5 = 25 blocks in the image 610, and the object extraction unit 111 performs object extraction processing in units of each block. Similarly, the DNN processing unit 110 divides the camera image 600 into 4×4 = 16 blocks in the image 620, divides the camera image 600 into 3×3 = 9 blocks in the image 630, divides the camera image 600 into 2×2 = 4 blocks in the image 640, and divides the camera image 600 into 1×1 = 1 block in the image 650 to perform object extraction processing.

[0139] Therefore, Figure 9 The number of times the DNN constituting the object extraction unit 111 in the first layer to the fifth layer shown performs object extraction processing is 25 + 16 + 9 + 4 + 1 = 55 times. Furthermore, regarding the structure of the DNN of the object extraction unit 111, as long as an object can be extracted from the camera image 600, it is not limited to the structure used in this embodiment that extracts small objects first and then large objects. In addition, regarding the number of block divisions and the number of layers when performing DNN operations in this case, as long as the accuracy required for realizing autonomous driving can be ensured, it is not limited to the Figure 9 example.

[0140] On the other hand, for example, in the camera image 600 from the camera 200, objects such as the hood 601 of the own vehicle and the sky 602 may be reflected as shown in Figure 9 . These objects are objects that do not need to be extracted from the camera image 600 when controlling the vehicle. Therefore, the object extraction process in the block area that only reflects unnecessary objects can also be reduced by specifying the block area where the object extraction unit 111 executes the DNN.

[0141] Figure 9 In, for example, in the case of a vehicle traveling only on a flat road with good visibility, the objects unnecessary for autonomous driving (clouds, hood) are always in the same position in the camera image 600. In this case, the object extraction process may not be performed in the block area of this part. The object extraction unit 111 performs the object extraction process only in the area designated as the block area where the DNN is executed.

[0142] InFigure 9 In this case, the area where the object extraction process is not performed is defined as the exclusion area, and the area where the object extraction process is executed is defined as the block area.

[0143] In the image 610 of the first layer of the object extraction unit 111, the area in the camera image 600 that only shows the sky is defined as the exclusion area 611-A, and the area in the camera image 600 that only shows the hood is defined as the exclusion area 611-B. Furthermore, in the case of the sky and the hood where the exclusion area is not specified, it is represented by the symbol "611" after omitting the symbol after "-". The symbols of other components are the same.

[0144] And the area between the exclusion areas 611-A and 611-B is set as the block area 612. That is, in the first layer, the object extraction unit 111 only needs to perform object extraction under DNN within the block area 612 in the third and fourth rows from the top in the image 610.

[0145] In the image 620 of the second layer, the object extraction unit 111 defines the area in the camera image 600 that only shows the sky as the exclusion area 621-A, and the area in the camera image 600 that only shows the hood as the exclusion area 621-B.

[0146] And the area between the exclusion areas 621-A and 621-B is set as the block area 622. In the second layer, the object extraction unit 111 only needs to perform object extraction under DNN within the block area in the second and third rows from the top in the image 620.

[0147] In the image 630 of the third layer, the object extraction unit 111 defines the area in the camera image 600 that only shows the sky as the exclusion area 631-A. And the area other than the exclusion area 631-A is set as the block area 632. In the third layer, the object extraction unit 111 only needs to perform object extraction under DNN within the block area 632 in the second and third rows from the top in the image 630.

[0148] In the image 640 of the fourth layer and the image 620 of the fifth layer, all areas are set as the block areas 642 and 652, and object extraction under DNN is performed in all blocks.

[0149] In this way, by presetting the exclusion area 611 where the object extraction process is not performed in the block area determination unit 124, the number of DNN operations becomes 10 + 8 + 6 + 4 + 1 = 29 times. Compared with the case of performing object extraction in all block areas of the camera image 600, 26 DNN operations can be reduced.

[0150] In the present embodiment, by using a fixed exclusion area in the camera image 600 to reduce the processing of the DNN of the object extraction unit 111, for example, it is possible to reduce the power consumption of the arithmetic device such as the FPGA 170 or GPU and suppress the heat generation. As a result, the travelable distance of the electric vehicle increases, and the cooling cost of the arithmetic device such as the FPGA 170 is reduced.

[0151] Furthermore, in the present embodiment, for the sake of simplicity of explanation, the values of the number of layers and the number of blocks of the DNN in Figure 9 are used as described above, but in the DNN used in actual object extraction, the camera image 600 as external information is divided into thousands to tens of thousands of blocks, and the number of layers is also large. Therefore, the actual number of operations is much larger than the example shown above, and the effect of reducing the amount of operations by reducing the block area to be executed is also extremely large.

[0152] Furthermore, in the present embodiment, external information is obtained from the camera 200, but it is not limited thereto. As the external information, for example, sensors such as LiDAR, RADAR, or far-infrared cameras that can obtain the distance to an object and the type of the object may be used. In addition, the sensors can be used alone or in combination of multiple types.

[0153] In addition, the block area determination unit 124 determines the block area to be executed by the DNN of the object extraction unit 111, but layer deletion can also be performed as in the first, second, third, fourth, fifth, and sixth embodiments. In this case, in addition to reducing the number of blocks used in the object extraction unit 111, the number of layers of the DNN is also reduced, thereby further reducing the power consumption.

[0154] Embodiment 8

[0155] Next, Embodiment 8 of the present invention will be described. Figure 10 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of the present embodiment. Figure 10 In, the same reference numerals and symbols are given to the same constituent elements as Figure 8 and, and unless otherwise specified, they are regarded as having the same or similar functions and the description thereof is omitted.

[0156] The present embodiment shows an example in which a processing condition determination unit 122 and sensors are added to the seventh embodiment, and the block area determination unit 124 in the FPGA 170 is moved to the memory 20. Other configurations are the same as those in the seventh embodiment.

[0157] Figure 10In [the description], since the processing condition determination unit 122 newly added in Embodiment 7 uses information indicating the driving state from sensors of a running vehicle, etc. as input to determine the current driving environment. The block area determination unit 124 dynamically changes the block area where DNN is executed in the object extraction process according to the driving environment determined by the processing condition determination unit 122 ( Figure 9 612, 622, 632, 642, 652 of [the reference], the same hereinafter).

[0158] In the present embodiment, as sensors for detecting the driving state of the vehicle, similar to Embodiment 2, an example is shown in which the vehicle speed sensor 30 and the illuminance sensor 40 are connected to the information processing device 100, but it is not limited to these.

[0159] Similar to Embodiment 2, the processing condition determination unit 122 of the information processing device 100 determines the driving environment based on information from the vehicle speed sensor 30 and the illuminance sensor 40 and outputs it to the block area determination unit 124. The block area determination unit 124 determines the block area (or exclusion area) used in the DNN in the form of processing conditions according to the input driving environment and instructs the object extraction unit 111.

[0160] In the example shown in Embodiment 7, since the block area is a fixed value, the block area determination unit 124 is configured in the FPGA 170. In Embodiment 8, the block area determination unit 124 dynamically changes the block area (or exclusion area). Therefore, the block area determination unit 124 is loaded into the memory 20 in the form of a program for the processor 10 to execute.

[0161] The example shown in Embodiment 7 is an example of statically determining the block area where DNN is executed in a restricted driving environment. However, in the autonomous driving of a vehicle, the driving environment is diverse and complex, and the accuracy of object extraction from image data is also changing all the time. Therefore, it is more desirable to be able to dynamically change the block area (or exclusion area) where DNN is executed according to the driving environment.

[0162] Next, the processing of the processing condition determination unit 122 will be described using a specific example.

[0163] As an example, an example is shown in which the vehicle speed detected by the vehicle speed sensor 30 is used as information indicating the driving state. If the vehicle speed detected by the vehicle speed sensor 30 is below a preset vehicle speed threshold, the processing condition determination unit 122 determines that the current driving environment is congested.

[0164] When the vehicle is moving at an extremely low speed (e.g., 5 km / h) in traffic congestion, the change in the image input from the camera 200 is small. Therefore, the object extraction unit 111 only needs to have the accuracy of object extraction capable of tracking the vehicle in front of its own vehicle, and it is not necessary to perform object extraction in all areas of the image from the camera 200.

[0165] The block area determination unit 124 determines the processing conditions according to the driving environment determined by the processing condition determination unit 122. The block area determination unit 124 determines Figure 9 the exclusion areas 611, 621, 631 where object extraction is not performed and the block areas 612, 622, 632, 642, 652 where object extraction is performed by means of DNN as the processing conditions in such a way as to reduce the computational load of the DNN while ensuring the accuracy of object extraction required for autonomous driving.

[0166] The object extraction unit 111 performs object extraction only within the block areas 612, 622, 632, 642, 652 determined by the block area determination unit 124. Furthermore, hereinafter, the block areas and the exclusion areas are sometimes simply referred to as such.

[0167] In this way, in the eighth embodiment, the processing condition determination unit 122 for determining the external environment (driving environment) according to the information indicating the driving state is added to the configuration of the seventh embodiment, and the determination of the processing conditions suitable for the driving environment is performed. Thus, compared with the seventh embodiment, the information processing apparatus 100 of this embodiment can dynamically change the accuracy of object extraction, the power consumption, and the heat generation amount of the FPGA 170 according to the driving environment.

[0168] As a result, compared with the seventh embodiment, the driving distance of the electric vehicle is increased and extended, and a reduction in the cooling cost of the FPGA 170 can be expected. Furthermore, the method for determining traffic congestion in the above description is not the vehicle speed sensor 30, and it can also be traffic congestion information such as image information, radio, and Internet that knows the driving speed.

[0169] In addition, using the illuminance sensor 40 as the information indicating the driving state, if the surrounding brightness (illuminance) is below a preset illuminance threshold, the processing condition determination unit 122 determines that the driving environment is inside a tunnel.

[0170] Figure 11 FIG. is an example showing the processing of performing DNN by the object extraction unit 111 in the camera image 600 from the camera 200 during tunnel driving. Inside the tunnel, the surroundings are covered by walls, so the processing of object extraction at the left and right ends in the camera image 600 may not be performed.

[0171] If the determination result from the processing condition determination unit 122 is that it is inside a tunnel, the block area determination unit 124 determines Figure 11An exclusion area 603 where object extraction is not performed, as shown. Regarding the method of determining the current driving environment as a tunnel, a device such as GPS that can determine the current position can be used, etc.

[0172] Furthermore, the block area determination unit 124 may also instruct the object extraction unit 111 to use an area with little or no change between frames of the camera image 600 as the exclusion area 603.

[0173] In addition, an in-vehicle camera (not shown) may be used as information indicating the driving state. In this case, the processing condition determination unit 122 uses the image data of the in-vehicle camera to detect whether the driver is operating the steering wheel. If the driver is operating the steering wheel, it is determined that autonomous driving (AD) is not being used. When using AD, the vehicle is controlled only by the autonomous driving system, so object extraction with high precision such as small objects needs to be recognized. In the case where AD is not used, the driver can know external information, so it is sufficient to maintain the accuracy of object extraction that can achieve driving assistance.

[0174] The block area determination unit 124 determines the block area where DNN is executed in the object extraction unit 111 according to the driving environment determined by the processing condition determination unit 122. Alternatively, the block area determination unit 124 may also determine an exclusion area 611, etc. where DNN is not executed. In addition, the method of determining whether AD is being used is not limited to the in-vehicle camera. Data such as a sensor that detects the operation of the steering wheel, a device that can determine whether the driver is in the driver's seat, or the presence or absence of an action plan for the autonomous driving vehicle generated by the action plan unit 130 can also be used to determine whether AD is being used.

[0175] Furthermore, the processing condition determination unit 122 may use any one of the information indicating the driving state to determine the processing conditions, or may use multiple pieces of information to determine the processing conditions.

[0176] Embodiment 9

[0177] Next, Embodiment 9 of the present invention will be described. Figure 12 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of the present embodiment. Figure 12 Among them, the same components as Figure 10 are given the same names and symbols. Unless otherwise specified, they are regarded as having the same or similar functions and the description is omitted.

[0178] In this embodiment, an example is shown in which a processing condition determination unit 122 and a load detection device 50 are added to the above-described Embodiment 7, and the block area determination unit 124 in the FPGA 170 is moved to the memory 20. Other configurations are the same as those in Embodiment 7.

[0179] In the said Embodiment 8, the processing condition determination unit 122 determines the current driving environment with information indicating the driving state from sensors during driving as input. In this embodiment, the processing condition determination unit 122 determines the load state of the current FPGA 170 with the operation load of the information processing device 100 as input. The block region determination unit 124 determines the block region of the DNN used by the object extraction unit 111 in object extraction according to the load state determined by the processing condition determination unit 122.

[0180] Figure 12 In, the load detection device 50 newly added from Embodiment 7 detects the operation load of the FPGA 170 and inputs it to the processing condition determination unit 122. Furthermore, the load detection device 50 may include sensors that detect the power consumption, supply current, etc. of the FPGA 170 as the operation load, for example.

[0181] In addition, in the example shown in Embodiment 7, since the block region (or excluded region) for executing the DNN is a fixed value, the block region determination unit 124 is configured in the FPGA 170. In Embodiment 9, the block region determination unit 124 dynamically changes the block region (processing condition), and thus is loaded into the memory 20 in the form of a program for the processor 10 to execute.

[0182] Next, the processing of the processing condition determination unit 122 will be described using a specific example.

[0183] As an example, the processing condition determination unit 122 calculates the load rate of the FPGA 170 as the operation load and compares it with a specified load rate threshold. Furthermore, regarding the load rate, an example of using the value (%) obtained by dividing the power consumption detected by the load detection device 50 by the specified maximum power is shown.

[0184] When the load rate of the FPGA 170 is higher than the load rate threshold, the processing condition determination unit 122 determines to reduce the operation load of the FPGA 170. The block region determination unit 124 determines the excluded region where object extraction is not performed in the object extraction unit 111 according to the determination result from the processing condition determination unit 122 so that the load rate of the FPGA 170 becomes below the load rate threshold.

[0185] Furthermore, the block region determination unit 124 may also instruct the object extraction unit 111 to use the region with no change (or little change) between frames of the camera image 600 as the excluded region 603.

[0186] Thus, by adding the processing condition determination unit 122 that determines the load state according to the operation load, the load state corresponding to the load rate of the FPGA 170 included in the information processing apparatus 100 can be determined, and the execution layer determination unit 121 can dynamically change the processing conditions according to the load state. Accordingly, compared with the seventh embodiment, the information processing apparatus 100 can dynamically reduce the power consumption and suppress the heat generation amount of the FPGA 170.

[0187] Furthermore, in the information processing apparatus 100 of the present embodiment, as described above, it is possible to suppress the occurrence of an abnormality in the FPGA 170 due to thermal runaway or the like and prevent the reduction of the calculation accuracy. Accordingly, compared with the seventh embodiment, it is possible to increase the travelable distance of the electric vehicle and reduce the cooling cost of the FPGA 170 or the like.

[0188] In addition, in the present embodiment, the load rate of the FPGA 170 as the operation load is not limited to the power consumption. A value associated with the operation load such as the temperature of the FPGA 170 may be used. Further, the above operation load may include not only the operation load of the FPGA 170 but also the operation load of the processor 10.

[0189] Furthermore, the processing condition determination unit 122 may determine the load state using any one of the operation loads in the operation load, or may determine the load state using a plurality of operation loads.

[0190] Embodiment 10

[0191] Next, Embodiment 10 of the present invention will be described. Figure 13 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of the present embodiment. Figure 13 Among them, the same reference numerals and symbols are given to the same constituent elements as Figure 10 , Figure 12 . As long as there is no particular description, they are regarded as having the same or similar functions and the description thereof is omitted.

[0192] In the present embodiment, an example in which the configuration of the ninth embodiment is added to the configuration of the eighth embodiment is shown. Other configurations are the same as those of the eighth embodiment.

[0193] In the eighth embodiment, an example in which the current driving environment is determined by using information indicating the driving state from sensors or the like during driving as an input is shown, and in the ninth embodiment, an example in which the current load state is determined by using the operation load of the information processing apparatus 100 as an input is shown. In the present embodiment, an example in which the eighth embodiment is combined with the ninth embodiment is shown.

[0194] The processing condition determination unit 122 uses both the information indicating the driving state during driving and the operation load of the FPGA 170 as inputs to determine the driving environment and the load state, and outputs the determination result to the block area determination unit 124. The block area determination unit 124 determines the block area (processing condition) of the DNN used in object extraction based on the determination result from the processing condition determination unit 122. Furthermore, the block area determination unit 124 only needs to determine the block area or the exclusion area as described above.

[0195] As described above, in the information processing apparatus 100 of the present embodiment, the driving environment and the load state are determined in the processing condition determination unit 122 based on the information indicating the driving state and the operation load of the FPGA 170. Therefore, compared with Embodiment 8 and Embodiment 9, it is possible to dynamically change the object extraction accuracy, power consumption, and heat generation reduction.

[0196] As a result, the information processing apparatus 100 of the present embodiment can expect an effect of combining the advantages of Embodiment 8 and Embodiment 9. That is, it is possible to maintain the load of the FPGA 170 in a normal state while maintaining an appropriate object extraction accuracy in various driving environments.

[0197] Furthermore, the processing condition determination unit 122 can use either the information indicating the driving state or the operation load to determine the processing conditions, or can use multiple pieces of information to determine the processing conditions.

[0198] Embodiment 11

[0199] Next, Embodiment 11 of the present invention will be described. Figure 14 FIG. is a block diagram showing an example of the configuration of an autonomous driving system using the information processing apparatus 100 of the present embodiment. Figure 14 Among them, the same components as Figure 13 are given the same names and symbols, and unless otherwise specified, they are regarded as having the same or similar functions and the description thereof is omitted.

[0200] This embodiment is obtained by adding a LiDAR 400, a sensor identification unit 140, and an identification accuracy determination unit 150 to the components of the above-described Embodiment 10. Other configurations are the same as those of Embodiment 10.

[0201] Figure 14 Among them, the newly added sensor identification unit 140 in Embodiment 10 processes the information (distance and azimuth) from the LiDAR 400 to identify an object. In addition, the identification accuracy determination unit 150 determines the difference between the object extraction result from the object extraction unit 111 and the object identification result from the sensor identification unit 140, thereby determining whether the object extraction by the object extraction unit 111 maintains the required accuracy, and outputs the determination result to the processing condition determination unit 122.

[0202] Subsequently, the reference value that is a condition for reducing the operation load is updated, thereby preventing excessive operation reduction such as insufficient accuracy of object extraction caused by the reduction of the operation load.

[0203] Subsequently, the processing condition determination unit 122 updates the reference value that is a condition for reducing the operation load in the object extraction unit 111 based on the determination result from the recognition accuracy determination unit 150, the driving state, and the operation load, thereby preventing excessive operation omission such as insufficient accuracy of object extraction caused by the reduction of the operation load.

[0204] In the seventh, eighth, ninth, and tenth embodiments, the block area for performing object extraction by means of DNN based on the information indicating the driving state and the operation load is static. To maintain the accuracy of object extraction required for autonomous driving, it is desirable for the reference value to be dynamic.

[0205] Next, a specific example is used to describe the processing of the recognition accuracy determination unit 150.

[0206] As an example, image data from the camera 200 is input. The number of objects extracted by the object extraction unit 111 is 5, and the number of objects recognized by the sensor recognition unit 140 based on the information from the LiDAR 400 is 7. The number of objects extracted by the object extraction unit 111 is less than the number of objects recognized by the sensor recognition unit 140.

[0207] In this case, the objects extracted by the sensor recognition unit 140 are not extracted by the object extraction unit 111, so it is determined that the object extraction unit 111 does not maintain the accuracy of object extraction required for autonomous driving.

[0208] The recognition accuracy determination unit 150 outputs a command to change the processing conditions in a manner that improves the accuracy of object extraction to the processing condition determination unit 122 based on the information from the object extraction unit 111 and the sensor recognition unit 140, and the processing condition determination unit 122 changes the reference value for determining the processing conditions.

[0209] Regarding the reference value of this embodiment, for example, when it is set as a value for correcting the load rate threshold, when the processing condition determination unit 122 receives a command to improve the accuracy of object extraction from the recognition accuracy determination unit 150, the reference value is set to +5% to increase the load rate threshold. In addition, the processing condition determination unit 122 notifies the block area determination unit 124 of the fact that the load rate threshold has been increased.

[0210] When the block area determination unit 124 receives a notification of an increase in the load rate threshold, it adds a prescribed value to the block area of the DNN currently in use in the object extraction unit 111 and instructs the object extraction unit 111 of the FPGA 170. The prescribed value for the block area determination unit 124 is set to 1, for example.

[0211] In this way, the block area determination unit 124 can increase the number of block areas of the DNN used in the object extraction unit 111 according to a request from the recognition accuracy determination unit 150 and improve the object extraction accuracy.

[0212] However, when the driving environment is congested or at night, the effect of increasing the block area of the DNN used in the object extraction unit 111 is low, so the block area determination unit 124 can cancel the notification of the increase in the load rate threshold.

[0213] In this way, by adding the sensor recognition unit 140 that uses the information from the LiDAR 400 to identify objects and the recognition accuracy determination unit 150 that determines the object extraction accuracy of the object extraction unit 111 and the sensor recognition unit 140, it is possible to prevent excessive reduction of the arithmetic processing (layer) with insufficient accuracy of object extraction required for autonomous driving in the object extraction unit 111.

[0214] As a result, the information processing device 100 of this embodiment can maintain a higher-precision processing than in Embodiments 7, 8, 9, and 10. As a result, the information processing device 100 of this embodiment can reduce the power consumption of the FPGA 170 compared to Embodiments 7, 8, 9, and 10.

[0215] Furthermore, in this embodiment, the sensor used for determining the object extraction accuracy is the Lidar 400, but it may be any sensor such as a camera, RADAR, far-infrared camera, etc. that can distinguish the distance to an object and the type of the object and is different from the sensor that outputs the external information input to the object extraction unit 111, and is not limited to LiDAR. In addition, the sensors can be used alone or in combination.

[0216] In addition, in this embodiment, as a method for the recognition accuracy determination unit 150 to determine the object extraction accuracy, it is a static process of determining whether the result of the object extraction unit 111 is consistent with the result of the sensor recognition unit 140, but the object extraction accuracy determination criterion can also be dynamically changed by making the recognition accuracy determination unit 150 have a learning function.

[0217] Embodiment 12

[0218] Next, Embodiment 12 of the present invention will be described. Figure 15 It is a block diagram showing an example of the configuration of an autonomous driving system using the information processing device 100 of this embodiment.Figure 15 In this, the same constituent elements as Figure 13 are given the same names and symbols, and as long as there is no special explanation, they are regarded as having the same or similar functions and the explanations are omitted.

[0219] In this embodiment, an example is shown in which an additional information determination unit 123 is added to the processing content control unit 120 of the embodiment 10 shown in Figure 13 , and an external information conversion unit 112 is added to the DNN processing unit 110 to change the number of pixels (or resolution) of the external image input from the camera 200 according to the driving environment and load state. Other configurations are the same as those of the embodiment 10.

[0220] Figure 15 In this, the additional information determination unit 123 and the external information conversion unit 112 newly added from the embodiment 10 are described. The additional information determination unit 123 determines the number of pixels (resolution) of the external image acquired by the camera 200 to be input to the object extraction unit 111 according to the result from the processing condition determination unit 122 and instructs the external information conversion unit 112 of the FPGA 170.

[0221] The external information conversion unit 112 converts the external image acquired by the camera 200 into a format usable in the DNN according to the instruction from the additional information determination unit 123. Thus, the object extraction unit 111 performs object extraction with the number of pixels corresponding to the processing condition (block area or exclusion area).

[0222] Next, the processing in the external information conversion unit 112 will be described using a specific example.

[0223] As an example, the camera 200 outputs an image of 2 million pixels. In addition, the additional information determination unit 123 determines that the number of pixels required for object extraction is 250,000 pixels according to the determination result of the processing condition determination unit 122. In this case, the external information conversion unit 112 converts the 2 million pixel image output from the camera 200 to become 250,000 pixels. The object extraction unit 111 uses the 250,000 pixel image output from this external information conversion unit to perform object extraction.

[0224] In this way, the information processing apparatus 100 of this embodiment dynamically changes the number of pixels of the input image according to the determination result from the processing condition determination unit 122, and thus can use an image with the number of pixels corresponding to the block area where the DNN is executed in the object extraction unit 111. Therefore, in addition to reducing the number of block areas of the DNN, it is also possible to reduce the number of pixels of the image for which the object extraction unit 111 performs object extraction, thereby reducing the amount of computation.

[0225] Thus, compared with the information processing apparatuses 100 of the above-described Embodiments 7, 8, 9, and 10, the information processing apparatus 100 of the present embodiment can achieve power consumption reduction and heat generation suppression. As a result, compared with Embodiments 7, 8, 9, and 10, the information processing apparatus 100 of the present embodiment increases the travelable distance of the electric vehicle, and an improvement in the cooling cost of the FPGA 170 can be expected.

[0226] Furthermore, in the present embodiment, the information amount determination unit 123 changes the number of pixels from the camera 200, and if the external information output is LiDAR or RADAR, the information amount of the external information such as the determination of the number of point clouds may be used, and is not limited to the number of pixels.

[0227] <Conclusion>

[0228] As described above, the information processing apparatus 100 of the above-described Embodiments 1 to 6 (7 to 12) may be configured as follows.

[0229] (1) An information processing apparatus (100) having a processor (10), a memory (20), and an arithmetic unit (FPGA 170) that performs arithmetic operations using an inference model, the information processing apparatus (100) being characterized by including: a DNN processing unit (110) that receives external information (output of the camera 200) and extracts an external object from the external information using the inference model; and a processing content control unit (120) that controls the processing content of the DNN processing unit (110), the DNN processing unit (110) including an object extraction unit (111), the object extraction unit (111) performing the inference model using a deep neural network having a layer with a plurality of neurons, the processing content control unit (120) including an execution layer determination unit (121), the execution layer determination unit (121) determining the layer used in the object extraction unit (111).

[0230] With the above configuration, the number of DNN layers (fixed values) used by the object extraction unit 111 is reduced, thereby enabling power consumption reduction and heat generation suppression of an arithmetic unit such as the FPGA 170 or GPU. As a result, the travelable distance of the electric vehicle increases, and the cooling cost of the arithmetic unit can be reduced.

[0231] (2) The information processing device according to the above (1), characterized in that the processing content control unit (120) includes a processing condition determination unit (122), the processing condition determination unit (122) receives external state information (driving state) and compares a preset threshold value with the value of the external state information to determine the external environment, and the execution layer decision unit (121) determines the number of layers used in the DNN processing unit (110) according to the determination result of the processing condition determination unit (122) and outputs it to the DNN processing unit (110).

[0232] With the above configuration, a sensor for additionally detecting the driving state and a processing condition determination unit 122 for determining the driving environment of the vehicle according to the information indicating the driving state are added, and the execution layer decision unit 121 makes a decision on the processing conditions suitable for the driving environment. Thus, compared with the above (1), the information processing device 100 can dynamically change the processing conditions that affect the accuracy of object extraction and the power consumption and heat generation of the arithmetic device (FPGA 170) according to the driving environment.

[0233] As a result, compared with the above (1), the available driving distance of the electric vehicle increases, and a reduction in the cooling cost of the arithmetic device (FPGA 170) can be expected.

[0234] (3) The information processing device according to the above (1), characterized in that it further has a load detection device (50) for detecting the arithmetic load of the arithmetic device (170), the processing content control unit (120) has a processing condition determination unit (122), the processing condition determination unit (122) receives the arithmetic load from the load detection device (50) and compares a preset load threshold value with the arithmetic load to determine the load state, and the execution layer decision unit (121) determines the number of layers used in the DNN processing unit (110) according to the determination result of the processing condition determination unit (122).

[0235] With the above configuration, the load state corresponding to the load of the FPGA 170 included in the information processing device 100 can be determined, and the execution layer decision unit 121 can dynamically change the processing conditions according to the load. Thus, compared with the above (1), the information processing device 100 can dynamically reduce the power consumption of the FPGA 170 and suppress the heat generation.

[0236] Furthermore, according to the above, the information processing device 100 of the present embodiment can suppress the occurrence of abnormalities in the FPGA 170 due to thermal runaway or the like and prevent the reduction of the arithmetic accuracy of the arithmetic device. Thus, compared with the first embodiment, the available driving distance of the electric vehicle can be increased, and the cooling cost of the FPGA 170 and the like can be reduced.

[0237] (4) The information processing apparatus according to (2) above, characterized in that

[0238] it further has a load detection device (50) for detecting the operation load of the arithmetic device (170), the processing condition determination unit (122) receives the operation load from the load detection device (50), compares the preset load threshold with the operation load to determine the load state, and outputs the determination result of the external environment and the determination result of the load state to the execution layer decision unit (121), and the execution layer decision unit (121) determines the number of the layers used in the DNN processing unit (110) according to the determination result of the processing condition determination unit (122).

[0239] With the above configuration, in the processing condition determination unit 122, the driving environment and the load state are determined based on the information indicating the driving state and the operation load of the FPGA 170. Thus, compared with (2) and (3) above, it is possible to dynamically change the accuracy of object extraction and reduce power consumption and heat generation. As a result, it is possible to expect the effect of combining the advantages of (2) and (3) above. That is, it is possible to keep the load of the FPGA 170 in a normal state while maintaining an appropriate object extraction accuracy in various driving environments.

[0240] (5) The information processing apparatus according to (4) above, characterized in that the processing content control unit (120) further has: a sensor identification unit (140) that receives sensor information (output of the LiDAR 400) for detecting an object and uses the sensor information to identify the object; and an identification accuracy determination unit (150) that calculates the identification accuracy by comparing the object identified by the sensor identification unit (140) with the object extracted by the DNN processing unit (110), and determines a decrease in the identification accuracy when the identification accuracy is below a preset accuracy threshold, and the processing condition determination unit (122) adds the determination result of the identification accuracy determination unit (150) to the determination result of the external environment and the determination result of the load state and outputs the result to the execution layer decision unit (121), and the execution layer decision unit (121) determines the number of the layers used in the DNN processing unit (110) according to the determination result of the processing condition determination unit (122).

[0241] With the above configuration, a sensor recognition unit 140 that additionally uses information from the LiDAR 400 to recognize an object, and a recognition accuracy determination unit 150 that determines the accuracy of object extraction by the object extraction unit 111 and the sensor recognition unit 140 are provided. Thereby, it is possible to prevent excessive reduction of the arithmetic processing (layer) where the accuracy of object extraction required for autonomous driving in the object extraction unit 111 is insufficient. As a result, the information processing device 100 can maintain a higher accuracy of processing than the above (1), (2), (3), and (4). Consequently, power consumption can be further reduced compared to the above (1), (2), (3), and (4).

[0242] (6) The information processing device according to (2) above, characterized in that the external information is image information, the processing content control unit (120) further includes an information amount determination unit (123) that determines the information amount of the image information based on the determination result of the processing condition determination unit (122), and the DNN processing unit (110) further includes an external information conversion unit (112). The external information conversion unit (112) changes the information amount of the image information used in the object extraction unit (111) according to the information amount determined by the information amount determination unit (123).

[0243] With the above configuration, the DNN processing unit 110 dynamically changes the number of pixels of the input image based on the determination result from the processing condition determination unit 122. Thereby, an image with the number of pixels corresponding to the number of execution layers of the DNN used in the object extraction unit 111 can be used. Therefore, in addition to reducing the execution layers of the DNN, the number of pixels of the image used in the object extraction unit 111 can also be reduced, further reducing the amount of computation. As a result, compared to the above (1), (2), (3), and (4), power consumption can be reduced and heat generation can be suppressed. Consequently, compared to the above (1), (2), (3), and (4), the driving range of the electric vehicle increases, and a reduction in the cooling cost of the FPGA 170 etc. can be expected.

[0244] Furthermore, the present invention includes various modification examples and is not limited to the above embodiments.

[0245] For example, the above embodiments are detailed descriptions for easily explaining the present invention and are not necessarily limited to having all the described configurations. In addition, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Moreover, the configuration of another embodiment can be added to the configuration of one embodiment. Additionally, addition, deletion, or replacement of other configurations can be individually or combinatorially applied to a part of the configuration of each embodiment.

[0246] In addition, some or all of the above-described components, functions, processing units, and processing methods can be implemented in hardware, for example, by designing them using integrated circuits. In addition, the above-described components and functions can also be implemented in software by a processor interpreting and executing a program that implements each function. Information such as programs, tables, and files that implement each function can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

[0247] In addition, the control lines and information lines shown are the parts considered necessary for explanation, and not all control lines and information lines are necessarily shown on the product. In fact, it can be considered that almost all components are interconnected.

[0248] <Supplementary>

[0249] As representative viewpoints of the present invention other than those described in the claims, the following viewpoints can be cited.

[0250] <7>

[0251] An information processing apparatus having a processor, a memory, and an arithmetic unit that performs arithmetic operations using an inference model, the information processing apparatus being characterized by including:

[0252] A DNN processing unit that receives external information and extracts an object in the external world from the external information using the inference model; and a processing content control unit that controls the processing content of the DNN processing unit. The DNN processing unit has an object extraction unit that includes the inference model. The inference model divides the external information into a plurality of block regions for extracting objects of different sizes, and extracts the object in the external world for each of the block regions using a deep neural network. The processing content control unit includes a block region determination unit that determines the block regions used in the object extraction unit.

[0253] <8>

[0254] The information processing apparatus according to <7> above, wherein the processing content control unit includes a processing condition determination unit that receives external state information and compares a preset threshold value with the value of the external state information to determine the external environment, and the block region determination unit determines the block regions used in the DNN processing unit based on the determination result of the processing condition determination unit and outputs them to the DNN processing unit.

[0255] <9>

[0256] The information processing apparatus according to <7> described above is characterized in that it further has a load detection device for detecting the operation load of the arithmetic device, the processing content control unit has a processing condition determination unit, the processing condition determination unit receives the operation load from the load detection device and compares a preset load threshold with the operation load to determine the load state, and the block area determination unit determines the block area used in the DNN processing unit according to the determination result of the processing condition determination unit.

[0257] <10>

[0258] The information processing apparatus according to <8> described above is characterized in that it further has a load detection device for detecting the operation load of the arithmetic device, the processing condition determination unit receives the operation load from the load detection device and compares a preset load threshold with the operation load to determine the load state, and outputs the determination result of the external environment and the determination result of the load state to the block area determination unit, and the block area determination unit determines the block area used in the DNN processing unit according to the determination result of the processing condition determination unit.

[0259] <11>

[0260] The information processing apparatus according to <10> described above is characterized in that the processing content control unit further has: a sensor recognition unit that receives sensor information of a detected object and uses the sensor information to recognize the object; and a recognition accuracy determination unit that compares the object recognized by the sensor recognition unit with the object extracted by the DNN processing unit to calculate the recognition accuracy, and determines a decrease in recognition accuracy when the recognition accuracy is below a preset accuracy threshold, and the processing condition determination unit outputs the determination result of the external environment, the determination result of the load state, and the determination result of the recognition accuracy determination unit to the block area determination unit, and the block area determination unit determines the block area used in the DNN processing unit according to the determination result of the processing condition determination unit.

[0261] <12>

[0262] The information processing apparatus according to <8> described above is characterized in that the external information is image information, the processing content control unit further has an information amount determination unit that determines the information amount of the image information according to the determination result of the processing condition determination unit, and the DNN processing unit further has an external information conversion unit that changes the information amount of the image information used in the object extraction unit according to the information amount determined by the information amount determination unit.

[0263] Symbol Explanation

[0264] 100… Information processing device

[0265] 110… DNN processing unit

[0266] 111… Object extraction unit

[0267] 112… External information conversion unit

[0268] 120… Processing content control unit

[0269] 121… Execution layer decision-making unit

[0270] 122… Processing condition determination unit

[0271] 123… Information volume determination unit

[0272] 124… Block area determination unit

[0273] 130… Action plan unit

[0274] 140… Sensor identification unit

[0275] 150… Identification accuracy determination unit

[0276] 200… Camera

[0277] 300… Vehicle control unit

[0278] 400… LiDAR

Claims

1. An information processing apparatus having a processor, a memory, and an arithmetic unit that performs operations using an inference model, the information processing apparatus being characterized by comprising: a load detection device that detects the arithmetic load of the arithmetic unit; a DNN processing unit that receives external information and extracts an object in the external world from the external information using the inference model; and a processing content control unit that controls the processing content of the DNN processing unit, wherein the DNN processing unit has an object extraction unit that executes the inference model using a deep neural network having a layer with a plurality of neurons, the processing content control unit includes: an execution layer determination unit that determines the layer used in the object extraction unit; a processing condition determination unit that receives external state information, compares a preset threshold value with the value of the external state information to determine the external environment, receives the arithmetic load from the load detection device, and compares a preset load threshold value with the arithmetic load to determine the load state; a sensor recognition unit that receives sensor information for detecting an object and recognizes the object using the sensor information; and a recognition accuracy determination unit that compares the object recognized by the sensor recognition unit with the object extracted by the DNN processing unit to calculate the recognition accuracy, and determines a decrease in the recognition accuracy when the recognition accuracy is below a preset accuracy threshold value, the processing condition determination unit adds the determination result of the external environment and the determination result of the load state to the determination result of the recognition accuracy determination unit and outputs the result to the execution layer determination unit, the execution layer determination unit determines the number of layers used in the DNN processing unit based on the determination result of the processing condition determination unit and outputs the result to the DNN processing unit.

2. The information processing apparatus according to claim 1, wherein, the external information is image information, the processing content control unit further has an information amount determination unit that determines the information amount of the image information based on the determination result of the processing condition determination unit, the DNN processing unit further has an external information conversion unit that changes the information amount of the image information used in the object extraction unit according to the information amount determined by the information amount determination unit.

Images