Cooling execution device, cooling execution method, cooling execution program, and vehicle

Abstract

The cooling execution device is provided with: an acquisition unit that acquires a detection result of an object related to the operation of a control device mounted on a vehicle for controlling the automatic driving of the vehicle; and an execution unit that executes cooling in the control device on the basis of the detection result.

Description

Technical Field

[0001] The present invention relates to a cooling execution device, a cooling execution method, a cooling execution program and a vehicle. Background Art

[0002] Japanese Patent Publication No. 2022-035198 describes a vehicle with an autonomous driving function. Summary of the invention

[0003] Means of solving problems

[0004] According to one embodiment of the present disclosure, there is provided: an acquisition unit, which acquires a detection result of an object related to the operation of a control device, wherein the control device is used to control the automatic driving of a vehicle and is mounted on the vehicle; and an execution unit, which executes cooling in the control device based on the detection result.

[0005] In the cooling execution device, the acquisition unit may extract a point used to represent the existence position of the object from an image of one frame of the object, and acquire motion information as the detection result at a frame rate of more than 100 frames per second, wherein the motion information represents the movement of the point used to represent the existence position of the object along a specified coordinate axis.

[0006] In any of the cooling execution devices, there may also be a prediction unit, which uses the detection result to predict the operation of the control device. The prediction unit uses a learning model generated by machine learning to predict the operation of the control device. The machine learning uses the detection result and the operating status of the control device when the detection result is obtained as learning data.

[0007] In any of the cooling execution devices, the prediction unit may further predict temperature changes of each of a plurality of locations in the control device, and the execution unit may control cooling of the locations in the control device.

[0008] In any of the cooling execution devices, it may also be provided with: an acquisition unit, which acquires the detection result of the object by the information processing device, and the information processing device outputs point information obtained by capturing the object as a point and identification information obtained by identifying the object based on the image of the object captured by multiple cameras facing corresponding directions, and associates the point information with the identification information; and an execution unit, which executes cooling of the information processing device based on the detection result acquired by the acquisition unit.

[0009] In any of the cooling execution devices, the execution unit may stop cooling the information processing device when a predetermined condition is satisfied.

[0010] In any of the cooling execution devices, the execution unit may stop cooling the information processing device when the predetermined condition that the object detected by the information processing device is no longer detected is satisfied.

[0011] In any of the cooling execution devices, the execution unit may stop cooling the information processing device when the predetermined condition that the object detected by the information processing device is moving out of a detection range is satisfied.

[0012] In any of the cooling execution devices, the execution unit may stop cooling the information processing device after a predetermined time has elapsed when a predetermined condition is satisfied.

[0013] In any of the cooling execution devices, there may also be a prediction unit, which predicts the working condition of the information processing device based on the detection result acquired by the acquisition unit, and the execution unit executes cooling of the information processing device based on the prediction result of the working condition of the information processing device by the prediction unit.

[0014] In any of the cooling execution devices, the prediction unit may predict a temperature change of the information processing device, and the execution unit may execute cooling of the information processing device using a cooling mechanism corresponding to a prediction result of the temperature change of the information processing device by the prediction unit.

[0015] In any of the cooling execution devices, the prediction unit may predict a period when the temperature of the information processing device is below a threshold value, and when the predetermined condition that the period is predicted by the prediction unit is satisfied, the execution unit stops cooling the information processing device.

[0016] In any of the cooling execution devices, the detection result acquired by the acquisition unit may be the point information.

[0017] In any of the cooling execution devices, it may also be provided with: a prediction unit, which predicts the operating condition and temperature change of the information processing device based on the detection result acquired by the acquisition unit; and an execution unit, which executes cooling of the information processing device based on the prediction result of the operating condition of the information processing device by the prediction unit, and the execution unit adjusts the cooling amount of the information processing device based on the prediction result of the temperature change of the information processing device by the prediction unit.

[0018] In any of the cooling execution devices, the prediction unit may further predict a peak value of the temperature of the information processing device, and the execution unit may execute cooling of the information processing device with a cooling amount corresponding to the peak value of the temperature predicted by the prediction unit.

[0019] In any of the cooling execution devices, the execution unit may execute cooling of the information processing device so that the cooling amount corresponding to the peak value of the temperature is reached before the temperature predicted by the prediction unit reaches the peak value.

[0020] In any of the cooling execution devices, the information processing device may be mounted on a mobile body, and when the detection result shows that the object is moving toward the mobile body, the prediction unit predicts a temperature rise in the information processing device, and when the prediction unit predicts a temperature rise in the information processing device, the execution unit increases the amount of cooling for the information processing device.

[0021] According to one embodiment of the present disclosure, a cooling execution method executed by a computer is provided. The cooling execution method may also be to obtain a detection result of an object related to the operation of a control device, the control device is used to control the automatic driving of the vehicle, is mounted on the vehicle, and based on the detection result, execute cooling in the control device.

[0022] According to one embodiment of the present disclosure, a cooling execution program for causing a computer to execute processing is provided. The cooling execution program may also be configured to obtain a detection result of an object related to the operation of a control device, the control device is used to control the automatic driving of the vehicle, is mounted on the vehicle, and based on the detection result, execute cooling in the control device.

[0023] According to one embodiment of the present disclosure, a vehicle is provided with a control device for controlling automatic driving, the vehicle comprising: a power unit, the power unit including an electric motor; an acquisition unit, the acquisition unit acquiring a detection result of an object related to the operation of the control device; and an execution unit, the execution unit executing cooling in the control device based on the detection result, the control device controlling various parts of the vehicle including the cooling by regenerative power generated by the electric motor when decelerating the vehicle in accordance with the detection result.

[0024] In any of the above vehicles, the control device may control the deceleration of the vehicle so that each component of the vehicle is controlled only by the regenerative electric power from the start of deceleration of the vehicle in accordance with the detection result until the vehicle stops.

[0025] In any of the above vehicles, the vehicle may include a brake actuator that performs a braking operation on the vehicle, and the control device may control the brake actuator using the regenerative power when decelerating the vehicle in accordance with the detection result.

[0026] In any of the above vehicles, the vehicle may include a steering actuator that performs a steering operation of the vehicle, and the control device may control the steering actuator using the regenerative electric power when decelerating the vehicle in accordance with the detection result.

[0027] In any of the vehicles, it may be that, as the detection result, the acquisition unit extracts a point used to represent the existence position of the object from an image of one frame of the object, and acquires motion information at a frame rate of more than 100 frames per second, wherein the motion information represents the movement of the point used to represent the existence position of the object along a specified coordinate axis.

[0028] In any of the vehicles, the vehicle may also include a prediction unit, which uses the detection result to predict the operation of the control device, and the prediction unit uses a learning model generated by machine learning to predict the operation of the control device, and the machine learning uses the detection result and the operating status of the control device when the detection result is obtained as learning data.

[0029] In any of the above vehicles, the prediction unit may further predict temperature changes of each of a plurality of locations in the control device, and the execution unit may control cooling of the locations in the control device.

[0030] It should be noted that the above summary of the invention does not list all the essential features of the present invention. In addition, sub-combinations of these feature groups may also constitute inventions. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 An example of the system 1 is schematically shown.

[0032] Figure 2 This is an explanatory diagram for explaining the learning phase in the system 1.

[0033] Figure 3 It is an explanatory diagram for explaining the cooling execution stage in the system 1.

[0034] Figure 4 An example of the SoCBox 400 and the cooling unit 600 is schematically shown.

[0035] Figure 5 An example of the SoCBox 400 and the cooling unit 600 is schematically shown.

[0036] Figure 6 An example of the SoCBox 400 and the cooling unit 600 is schematically shown.

[0037] Figure 7 A vehicle according to a modified example is schematically shown.

[0038] Figure 8 This is a schematic diagram showing an example of a vehicle equipped with a central brain.

[0039] Fig. 9 This is a first block diagram showing an example of the configuration of the information processing device.

[0040] Fig.10 This is a second block diagram showing an example of the configuration of the information processing device.

[0041] Fig.11 It is an explanatory diagram showing an example of point information output by MoPU.

[0042] Fig.12 This is a third block diagram showing an example of the structure of the information processing device.

[0043] Fig.13 This is a fourth block diagram showing an example of the structure of the information processing device.

[0044] Fig.14 This is an explanatory diagram showing an example of the relationship between point information and tag information.

[0045] Fig.15 It is an explanatory diagram showing a schematic structure of a vehicle.

[0046] Fig.16 : is a block diagram showing an example of the functional structure of the cooling execution device.

[0047] Fig.17 This is a fifth block diagram showing an example of the structure of the information processing device.

[0048] Fig.18 This is a sixth block diagram showing an example of the structure of the information processing device.

[0049] Fig.19 FIG. 1 is a diagram schematically showing coordinate detection of an object in time series.

[0050] Fig. 20 This is a seventh block diagram showing an example of the structure of the information processing device.

[0051] Fig.21 This is an explanatory diagram for explaining the image of an object captured by the event camera.

[0052] Fig. 22 This is an explanatory diagram schematically showing an example of the hardware configuration of a computer functioning as a management server, a SoCBox, an information processing device, or a cooling execution device. DETAILED DESCRIPTION

[0053] The present invention will be described below by way of the embodiments of the invention, but the following embodiments do not limit the invention involved in the claims. In addition, not all combinations of features described in the embodiments are essential to the solution of the present invention.

[0054] (First Embodiment)

[0055] When a system-on-chip (SoC) for autonomous driving performs high-level computing, heat generation becomes a problem. Therefore, in this embodiment, a cooling execution device is provided, which detects the processing work of the SoCBox and executes cooling of the SoCBox.

[0056] As an opportunity for the SoCBox to work, a moving object in the lane is sometimes detected. For example, when a moving object is detected in the lane during autonomous driving, the SoCBox sometimes performs calculations on the object to control the vehicle. However, since the SoCBox immediately becomes hot, it is difficult to perform high-level calculations in the vehicle (a problem faced by fully autonomous driving). Therefore, it is considered to predict the heat dissipation caused by the operation of the SoCBox by detecting moving objects in the lane, and to cool the SoCBox. For example, the heat dissipation of the SoCBox can be predicted by detecting moving objects, and cooling can be performed simultaneously with the heat dissipation to prevent the SoCBox from becoming hot, making it possible to perform high-level calculations in the vehicle.

[0057] By detecting moving objects, etc., the heat dissipation of the SoCBox is predicted, and cooling is performed simultaneously with the heat dissipation to prevent the SoCBox from becoming too hot, making it possible to perform advanced computing in the vehicle.

[0058] Figure 1 An example of the system 1 is schematically shown. The system 1 includes a management server 101 , a SoCBox 400 , a cooling execution device 500 , and a cooling unit 600 .

[0059] The SoCBox 400, the cooling actuator 500, and the cooling unit 600 are mounted on a vehicle. The SoCBox 400 uses the sensor values ​​of multiple sensors mounted on the vehicle to control the automatic driving of the vehicle. Since the automatic driving control of the vehicle is subjected to a very high processing load, the SoCBox 400 sometimes becomes very hot. When the SoCBox 400 becomes too hot, the operation of the SoCBox 400 may no longer be performed normally, and may have an adverse effect on the vehicle.

[0060] The cooling execution device 500 according to the present embodiment predicts, for example, the operation of the SoCBox 400, and starts cooling the SoCBox 400 based on the operation. For example, when a moving object that is a factor in the operation of the SoCBox 400 is detected, the cooling execution device 500 immediately starts cooling the SoCBox 400. By starting cooling before or at the same time as the start of heat generation due to the operation of the SoCBox 400, it is possible to reliably prevent the SoCBox 400 from becoming hot.

[0061] It should be noted that the cooling execution device 500 can predict the operation of the SoCBox 400 through artificial intelligence (AI). The operation of the SoCBox 400 can be learned by using data collected by the vehicle 200. For example, the management server 101 collects data from the vehicle 200 and performs learning. The subject that performs learning is not limited to the management server 101, and it can also be other devices.

[0062] The vehicle 200 is equipped with a SoCBox 400 and a MoPU (Motion Processing Unit) 217 ​​as a sensor. The SoCBox 400 controls the autonomous driving of the vehicle 200 using sensor values ​​of multiple sensors including the MoPU 217 mounted on the vehicle 200 or external information received from various types of servers 31 .

[0063] Here, the MoPU 217 may be built into a low-resolution camera (not shown) provided in the vehicle. The MoPU 217 outputs motion information indicating the motion of the captured object at a frame rate of, for example, 100 frames per second. The MoPU 217 outputs vector information indicating the motion of a point at which the object exists along a prescribed coordinate axis as motion information. That is, the motion information output from the MoPU 217 does not include information required to identify what the captured object is (for example, a person or an obstacle), but only includes information indicating the motion (movement direction and movement speed) of the center point (or center of gravity) of the object on the coordinate axis (x-axis, y-axis, z-axis).

[0064] The server 31 may be an example of an external device. Examples of various servers 31 include a server that provides traffic information and a server that provides weather information. The SoCBox 400 transmits sensor values ​​or external information including detection results of the MoPU 217 used in the control of autonomous driving, the operating status of the SoCBox 400 during control, and temperature changes of the SoCBox 400 during control to the management server 101.

[0065] The management server 101 performs learning using information received from one or more SoCBoxes 400 and a plurality of sensors. The management server 101 performs machine learning using the detection results of the MoPU 217 acquired by the SoCBox 400 and the operating conditions of the SoCBox 400 when the SoCBox 400 acquires the information as learning data, thereby generating a learning model that uses the information acquired by the SoCBox 400 as input and the operating conditions of the SoCBox 400 as output.

[0066] The vehicle 300 is a vehicle having a cooling function according to the present embodiment. The vehicle 300 is equipped with the MoPU 217 as a sensor, the SoCBox 400, the cooling execution device 500, and the cooling unit 600. The cooling execution device 500 can receive and store the learning model generated by the management server 101 from the management server 101.

[0067] The cooling execution device 500 can obtain sensor values ​​of multiple sensors mounted on the vehicle 300 including the detection results of the MoPU 217 from multiple sensors or the SoCBox 400, and when the operation of the SoCBox 400 is predicted, the cooling unit 600 starts to cool the SoCBox 400. Specifically, when a moving object or the like is detected as a detection result of the MoPU 217, the cooling execution device 500 predicts that the SoCBox 400 is operating, and the cooling unit starts to cool the SoCBox 400.

[0068] Furthermore, the cooling execution device 500 can predict the operation of the SoCBox 400 by acquiring sensor values ​​of a plurality of sensors mounted on the vehicle 300 including the detection results of the MoPU 217 from the plurality of sensors or the SoCBox 400 and inputting the acquired information into the learning model.

[0069] The SoCBox 400, the cooling execution device 500, the management server 101, and the server 31 may communicate through the network 20. The network 20 may include a vehicle network. The network 20 may include the Internet. The network 20 may include a local area network (LAN). The network 20 may include a mobile communication network. The mobile communication network may comply with any of the 5G (5th Generation) communication method, the LTE (Long Term Evolution) communication method, the 3G (3rd Generation) communication method, the 6G (6th Generation) communication method, and subsequent communication methods.

[0070] Figure 2 2 is an explanatory diagram for explaining the learning stage in the system 1. Here, as the sensor 210 mounted on the vehicle 200, a camera 211, a LiDAR (Light Detection And Ranging) 212, a millimeter wave sensor 213, an ultrasonic sensor 214, an IMU sensor 215, a GNSS (Global Navigation Satellite System) sensor 216, a MoPU 217, and a temperature sensor 218 are exemplified. The vehicle 200 does not have all of these sensors, but may have some of these sensors, or may have sensors other than these sensors.

[0071] The SoCBox 400 acquires sensor information from each sensor included in the sensor 210. In addition, the SoCBox 400 can perform communication via the network 20, and the SoCBox 400 receives external information from each of the plurality of servers 31 via the network 20. And, the SoCBox 400 performs the automatic driving control of the vehicle 200 using the acquired information.

[0072] As a work of the SoCBox 400, the SoCBox 400 records the computing power. The SoCBox 400 may record the computing power of the SoCBox 400 periodically or irregularly. The SoCBox 400 may record the sensor information received from the sensor 210, the external information received from the server 31, and the computing power of the SoCBox 400 when acquiring the information and performing the autonomous driving control, and transmit the same to the management server 101.

[0073] The management server 101 includes an information acquisition unit 102 , a model generation unit 104 , and a model provision unit 106 . The information acquisition unit 102 acquires various information. The management server 101 can receive information transmitted by the SoCBox 400 .

[0074] The model generation unit 104 performs machine learning using the information acquired by the information acquisition unit 102 to generate a learning model.

[0075] The model generation unit 104 can generate a learning model that takes the information acquired by the SoCBox 400 as input and the operating status of the SoCBox 400 as output by executing machine learning using the information acquired by the SoCBox 400 and the operating status of the SoCBox 400 when the SoCBox 400 acquired the information as learning data. Specifically, the model generation unit 104 generates a learning model that takes the detection result of the MoPU 217 as information acquired by the SoCBox 400 as input and outputs the status and change amount of the power calculation capability as the operating status of the SoCBox 400.

[0076] The model providing unit 106 provides the learning model generated by the model generating unit 104. The model providing unit 106 may transmit the learning model to the cooling execution device 500 mounted on the vehicle 300.

[0077] The system 1 may be configured to predict temperature changes of multiple parts of the SoCBox 400. In this case, the vehicle 200 may include multiple temperature sensors 218, and the multiple temperature sensors 218 measure temperature changes of multiple parts of the SoCBox 400. The SoCBox 400 may send sensor information received from the sensor 210, external information received from the server 31, and temperature changes measured by the multiple temperature sensors 218 when the information is acquired and the automatic driving control is performed to the management server 101. The model generation unit 104 performs machine learning by using the information acquired by the SoCBox 400, the temperature changes of multiple parts of the SoCBox 400 when the SoCBox 400 acquires the information, and the operating conditions of the SoCBox 400 as learning data, and generates a learning model that uses the information acquired by the SoCBox 400 as input and the temperature changes of multiple parts in the SoCBox 400 and the operating conditions of the SoCBox 400 as output.

[0078] Figure 33 is an explanatory diagram for explaining the cooling execution stage in the system 1. Here, as the sensor 310 mounted on the vehicle 300, a camera 311, a LiDAR 312, a millimeter wave sensor 313, an ultrasonic sensor 314, an IMU sensor 315, a GNSS sensor 316, a MoPU 317, and a temperature sensor 318 are exemplified. The vehicle 300 does not have all of these sensors, but may have some of these sensors, or may have sensors other than these sensors.

[0079] The cooling execution device 500 includes an information acquisition unit 502 , a cooling execution unit 504 , a model storage unit 506 , and a prediction unit 508 .

[0080] The information acquisition unit 502 acquires information acquired by the SoCBox 400. The information acquisition unit 502 acquires sensor information acquired by the SoCBox 400 from the sensor 310 from the sensor 310 or the SoCBox 400. For example, the information acquisition unit 502 may receive sensor information acquired by the SoCBox 400 from the sensor 310 from the SoCBox 400. The information acquisition unit 502 may receive the same sensor information from the sensor 310 as the sensor information acquired by the SoCBox 400 from the sensor 310. In this case, each of the sensors 310 may send the sensor information to the SoCBox 400 and the cooling execution device 500, respectively.

[0081] The cooling execution unit 504 starts cooling the SoCBox 400 based on the detection result of the MoPU 217 included in the sensor information. For example, when the detection result of the MoPU 217 indicates that a moving object is detected, the cooling execution unit 504 starts cooling the SoCBox 400.

[0082] The cooling execution unit 504 executes cooling of the SoCBox 400 using the cooling unit 600. The cooling unit 600 cools the SoCBox 400 by air cooling, water cooling, or liquid nitrogen cooling.

[0083] It should be noted that in this embodiment, the method of predicting the operation of SoCBox400 when a moving object is detected is described. However, it is not limited to this. The prediction unit 508 can also predict the operation of SoCBox400 through AI.

[0084] For example, the model storage unit 506 stores the learning model received from the management server 101. The prediction unit 508 predicts the operating status of the SoCBox 400 by inputting the detection result of the MoPU 217 acquired by the information acquisition unit 502 into the learning model stored in the model storage unit 506. Here, the learning model outputs the status and change amount of the power calculation capability of the SoCBox 400 as the operating status. In addition, the prediction unit 508 can predict and output the temperature changes of each of the plurality of parts in the SoCBox 400 together with the operating status.

[0085] The cooling execution unit 504 may start cooling the SoCBox 400 according to the working condition of the SoCBox 400 predicted by the AI. For example, when the condition and the amount of change of the power computing power of the SoCBox 400 predicted as the working condition exceed a predetermined threshold, the cooling execution unit 504 starts cooling the SoCBox 400. In addition, the cooling execution unit 504 may start cooling the SoCBox 400 according to the working condition of the SoCBox 400 predicted by the AI ​​and the temperature change of a part in the SoCBox 400. For example, when the condition and the amount of change of the power computing power of the SoCBox 400 predicted as the working condition exceed a predetermined threshold and the temperature change exceeds a predetermined threshold, the cooling execution unit 504 starts cooling the corresponding part in the SoCBox 400.

[0086] In addition, the cooling unit 600 may include multiple cooling mechanisms. For example, the cooling unit 600 includes multiple air cooling mechanisms. For example, the cooling unit 600 includes multiple water cooling mechanisms. For example, the cooling unit 600 includes multiple liquid nitrogen cooling mechanisms. The cooling unit 600 may include multiple of one or more air cooling mechanisms, one or more water cooling mechanisms, and one or more liquid nitrogen cooling mechanisms.

[0087] The plurality of cooling mechanisms may be configured to cool different parts of the SoCBox 400. The prediction unit 508 may predict the temperature changes of the plurality of parts of the SoCBox 400 using the information acquired by the information acquisition unit 502. The cooling execution unit 504 may start cooling the SoCBox 400 using a cooling mechanism selected from the plurality of cooling mechanisms that cool the plurality of parts of the SoCBox 400, based on the prediction result of the prediction unit 508.

[0088] The cooling execution unit 504 may use a cooling mechanism corresponding to the degree of the temperature of the SoCBox 400 predicted by the prediction unit 508 to execute cooling of the SoCBox 400. For example, the higher the temperature of the SoCBox 400, the more cooling mechanisms the cooling execution unit 504 uses to cool the SoCBox 400. As a specific example, when it is predicted that the temperature of the SoCBox 400 exceeds a first threshold, the cooling execution unit 504 starts cooling using one of the plurality of cooling mechanisms, and when it is predicted that the temperature of the SoCBox 400 still rises and exceeds a second threshold, the cooling execution unit 504 increases the number of cooling mechanisms to be used.

[0089] The higher the temperature of SoCBox 400 is, the more powerful cooling mechanism can be used by the cooling execution unit 504 to cool the SoCBox 400. For example, when it is predicted that the temperature of SoCBox 400 exceeds the first threshold, the cooling execution unit 504 starts cooling using the air cooling mechanism, when it is predicted that the temperature of SoCBox 400 still rises and exceeds the second threshold, the cooling execution unit 504 starts cooling using the water cooling mechanism, and when it is predicted that the temperature of SoCBox 400 still rises and exceeds the third threshold, the cooling execution unit 504 starts cooling using the liquid nitrogen cooling mechanism.

[0090] The SoCBox 400 may have a plurality of processing chips, and the plurality of processing chips may be respectively arranged at different positions of the SoCBox 400. A plurality of cooling mechanisms may be respectively arranged at positions corresponding to each of the plurality of processing chips.

[0091] For example, when the number of processing chips used changes according to the control conditions of autonomous driving, cooling is performed by a cooling mechanism corresponding to the processing chips being used, thereby enabling efficient cooling.

[0092] It should be noted that, in the present embodiment, a method of cooling the SoCBox 400 when a moving object is detected by the MoPU 217 is described. However, this is not limited to this. When a moving object is detected by the MoPU 217, regeneration of electric power based on the heat generated by the SoCBox 400 may also be started. For example, a Peltier element may be provided in the SoCBox 400, and when a moving object is detected by the MoPU 217, power generation based on the Peltier element using the heat generated by the SoCBox 400 may be started. By using the Peltier element, electric power can be obtained immediately when the SoCBox 400 generates heat, thereby improving energy efficiency.

[0093] Figure 4 An example of the SoCBox 400 and the cooling unit 600 is schematically shown. Figure 4 The case where the cooling unit 600 is composed of one cooling mechanism is exemplified. When the cooling execution device 500 detects a moving object, it can cool the entire SoCBox 400 by starting cooling by the cooling unit 600 .

[0094] Figure 5 An example of the SoCBox 400 and the cooling unit 600 is schematically shown. Figure 5 The example shows a case where the cooling unit 600 is composed of a plurality of cooling mechanisms for cooling a plurality of parts of the SoCBox 400. The cooling execution device 500 predicts the temperature changes of the plurality of parts of the SoCBox 400 by using the temperature sensor 318, and performs cooling using only the cooling mechanism corresponding to a certain part when it is predicted that a certain part starts to heat up or the temperature of a certain part exceeds a predetermined threshold, thereby achieving efficient cooling.

[0095] Figure 6 An example of the SoCBox 400 and the cooling unit 600 is schematically shown. Figure 6 The example shows a case where the cooling unit 600 is composed of two cooling mechanisms. The cooling execution device 500 predicts the temperature changes of the multiple parts of the SoCBox 400, and executes cooling using only the cooling mechanism corresponding to the part according to the prediction that a certain part starts to heat up or the temperature of a certain part exceeds a predetermined threshold, thereby achieving efficient cooling. In addition, as the temperature of the SoCBox 400 increases, the cooling execution device 500 increases the cooling mechanism used, that is, in this example, first starts to use one of the two cooling mechanisms for cooling, and when the temperature of the SoCBox 400 further increases, starts to use the other cooling mechanism for cooling, thereby making it possible to make the energy used for cooling more efficient.

[0096] [Modifications]

[0097] Modifications of the vehicle 300 of the above-described embodiment will be described below.

[0098] When the detection result of the MoPU 217 indicates that a moving object is detected, the cooling execution unit 504 uses the cooling unit 600 to execute cooling of the SoCBox 400 while decelerating the vehicle 300. And, as described below, the cooling execution device 500 and the cooling unit 600 are driven by the regenerative power (regenerative energy) generated when the vehicle 300 decelerates.

[0099] like Figure 7As shown, the vehicle 300 of this embodiment includes a power unit 704 for traveling including an electric motor 702, a battery 706 for supplying electric power to the electric motor 702, and a power control unit 708. It should be noted that the power unit 704 may include only the electric motor 702, or may include both the electric motor 702 and an internal combustion engine (engine).

[0100] The electric motor 702 can generate a driving force required for the vehicle 300 to travel, and can also function as a generator to generate regenerative electric power when the vehicle 300 is decelerating.

[0101] The power control unit 708 can adjust the power delivered from the battery 706 to the motor 702, and can deliver the regenerative power to the cooling actuator 500 and the cooling unit 600 when the vehicle is decelerating. Furthermore, the power control unit 708 can adjust the regenerative electric energy of the regenerative power generated by the motor 702 based on the instruction from the SoCBox 400, thereby adjusting the deceleration of the vehicle. In other words, by adjusting the deceleration of the vehicle 300, the regenerative electric energy can be adjusted.

[0102] In this embodiment, when the vehicle decelerates (indicates when a moving object is detected), regenerative power is generated, and the regenerative power can be supplied to the cooling actuator 500 and the cooling unit 600, thereby suppressing the power consumption of the battery 706. Furthermore, by adjusting the deceleration, all the power used in the cooling actuator 500 and the cooling unit 600 can be supplied by the regenerative power, and at least the cooling actuator 500 and the cooling unit 600 can be made not to use the power of the battery 706 during deceleration.

[0103] When there is a surplus of regenerative power, the power control unit 708 may also supply regenerative power to electrical equipment (such as the brake actuator 710 and the steering actuator 712 described later) other than the cooling actuator 500 and the cooling unit 600. In addition, when there is a surplus of regenerative power, the power control unit 708 may also charge the battery 706 using the surplus regenerative power.

[0104] like Figure 7 As shown, the vehicle 300 has a brake actuator 710 for performing braking operations during deceleration, and a steering actuator 712 for performing steering operations of the vehicle 300 based on the detection results of MoPU317, and during deceleration, the power used in the brake actuator 710 and the power used in the steering actuator 712 can be supplied by regenerative power.

[0105] For example, when the vehicle 300 detects a moving object on a lane while performing automatic driving, the steering actuator 712 can operate a steering mechanism (not shown) of the vehicle 300 to enable the vehicle 300 to avoid the object.

[0106] Furthermore, even if an object approaching the vehicle 300 is detected and an evasive action is not taken in time, a collision occurs, and the power supply from the battery 706 is lost, the brake actuator 710 can be operated by regenerative power to stop the vehicle safely.

[0107] Furthermore, before the vehicle 300 is stopped by performing a braking operation, the SoCBox 400 can control the deceleration of the vehicle 300 generated by the brake actuator 710 so that the power used in the control system, cooling system, and brake system, etc., such as at least the SoCBox 400 (an example of a control device), the cooling actuator 500, the cooling unit 600, the power control unit 708, and the brake actuator 710, can be supplied only by regenerative power. For example, when decelerating, if the power used in the control system, cooling system, and brake system, in other words, the system operating during deceleration is about to be insufficient due to the regenerative power at that time, the SoCBox 400 increases the deceleration to increase the regenerative electric energy.

[0108] It should be noted that the power control unit 708 can be operated so that the SoCBox 400 is cooled using the power of the battery 706 before regeneration is performed, and the SoCBox 400 is cooled using the regenerated power when regeneration is performed.

[0109] Furthermore, the SoCBox 400 can control the power control unit 708 to switch the supply destination of the regenerative power according to the remaining power of the battery 706. However, in this case, the cooling of the SoCBox 400 is given the highest priority. It should be noted that when the remaining power of the battery 706 is sufficient, the power of the battery 706 can also be used for cooling, etc.

[0110] (Second Embodiment)

[0111] Next, the second embodiment involved in the present embodiment is described. As an example, at least a portion of the information processing device involved in the present disclosure is mounted on the vehicle 100 to perform automatic driving control of the vehicle 100. In addition, the information processing device can realize automatic driving (Autonomous Driving) in real time based on data obtained through AI / multivariable analysis / goal seeking (Goal Seek) / strategy formulation / optimal probability solution / optimal speed solution / optimal route management / through input from multiple sensors at the edge based on level 6, and can provide a driving system adjusted based on the Delta optimal solution. Vehicle 100 is an example of an "object".

[0112] Here, "Level 6" refers to a level of autonomous driving, which is equivalent to a level higher than Level 5, which indicates fully autonomous driving. Although Level 5 indicates fully autonomous driving, it is a level equivalent to human driving, and there is still a probability of accidents. Level 6 is a level higher than Level 5, which is equivalent to a level with a lower probability of accidents than Level 5.

[0113] The computing power in level 6 is about 1,000 times that of level 5. Therefore, high-performance driving control that cannot be achieved in level 5 can be achieved.

[0114] Figure 8 1 is a schematic diagram showing an example of a vehicle 100 equipped with a central brain 15. A plurality of gateways are communicatively connected to the central brain 15. The central brain 15 is connected to an external cloud server via a gateway. The central brain 15 is configured to be able to access an external cloud server via a gateway. On the other hand, due to the presence of a gateway, the central brain 15 is configured to be inaccessible directly from the outside.

[0115] The central brain 15 outputs a request signal to the cloud server every predetermined time. Specifically, the central brain 15 outputs a request signal indicating an inquiry to the cloud server every nanosecond. As an example, the central brain 15 controls level 6 autonomous driving based on multiple information obtained via the gateway.

[0116] Fig. 9 1 is a first block diagram showing an example of the structure of the information processing device 10. The information processing device 10 includes an IPU (Image Processing Unit) 11, a MoPU (Motion Processing Unit) 12, a central brain 15, and a memory 16. The central brain 15 is configured to include a GNPU (Graphics Neural Network Processing Unit) 13 and a CPU (Central Processing Unit) 14.

[0117] The IPU 11 is built into an ultra-high resolution camera (not shown) provided in the vehicle 100. The IPU 11 performs prescribed image processing such as Bayer transformation, de-mosaicing, denoising, and sharpening on the image of the object existing around the vehicle 100 captured by the ultra-high resolution camera, and outputs the processed image of the object at a frame rate of, for example, 10 frames per second and a resolution of 12 million pixels. In addition, the IPU 11 outputs identification information obtained by identifying the captured object based on the image of the object captured by the ultra-high resolution camera. The identification information is the information required to identify what the captured object is (for example, a person or an obstacle). In this embodiment, as identification information, the IPU 11 outputs label information indicating the category of the captured object (for example, information indicating whether the captured object is a dog, a cat, or a bear). Furthermore, the IPU 11 outputs position information indicating the position of the captured object in the camera coordinate system of the ultra-high resolution camera. The image, label information, and position information output from the IPU 11 are provided to the central brain 15 and the memory 16. IPU 11 is an example of a "second processor" and the ultra-high-resolution camera is an example of a "second camera".

[0118] The MoPU 12 is built into another camera (not shown) different from the ultra-high resolution camera provided in the vehicle 100. The MoPU 12 outputs point information obtained by capturing the photographed object as a point at a frame rate of, for example, 100 frames per second, based on the image of the object photographed by the other camera facing the direction corresponding to the ultra-high resolution camera at a frame rate of 100 frames per second or more. The point information output from the MoPU 12 is provided to the central brain 15 and the memory 16. In this way, the image used by the MoPU 12 to output the point information and the image used by the IPU 11 to output the identification information refer to the images photographed by the other camera and the ultra-high resolution camera in the corresponding direction. Here, the "corresponding direction" refers to the direction in which the photographic range of the other camera overlaps with the photographic range of the ultra-high resolution camera. In the above case, the other camera photographs the object in the direction overlapping with the photographic range of the ultra-high resolution camera. It should be noted that the ultra-high resolution camera and the other camera photograph the object in the corresponding direction, for example, by pre-calculating the correspondence between the camera coordinate systems of the ultra-high resolution camera and the other camera.

[0119] For example, as point information, MoPU 12 outputs the coordinate values ​​of at least two coordinate axes in the three-dimensional orthogonal coordinate system of the point representing the existence position of the object. As an example, the coordinate value represents the center point (or center of gravity) of the object. In addition, MoPU12 outputs the coordinate value of the axis along the width direction (x-axis) (hereinafter referred to as "x-coordinate value") and the coordinate value of the axis along the height direction (y-axis) in the three-dimensional orthogonal coordinate system (hereinafter referred to as "y-coordinate value") as the coordinate values ​​of the two coordinate axes. It should be noted that the x-axis is the axis along the vehicle width direction of the vehicle 100, and the y-axis is the axis along the height direction of the vehicle 100.

[0120] According to the above structure, since the point information output by the MoPU 12 for one second includes the x-coordinate value and the y-coordinate value of more than 100 frames, the movement (moving direction and moving speed) of the object on the x-axis and the y-axis in the three-dimensional orthogonal coordinate system can be grasped based on the point information. In other words, the point information output by the MoPU 12 includes the position information indicating the position of the object in the three-dimensional orthogonal coordinate system and the motion information indicating the motion of the object.

[0121] As described above, the point information output from the MoPU 12 does not include information required for identifying what the photographed object is (e.g., a person or an obstacle), but only includes information indicating the movement (moving direction and moving speed) of the center point (or center of gravity) of the object on the x-axis and y-axis. In addition, since the point information output from the MoPU 12 does not include image information, the amount of data output to the central brain 15 and the memory 16 can be epoch-makingly reduced. The MoPU 12 is an example of a "first processor", and the other cameras are examples of a "first camera".

[0122] As described above, in this embodiment, the frame rate of the other camera with built-in MoPU 12 is greater than the frame rate of the ultra-high resolution camera with built-in IPU 11. Specifically, the frame rate of the other camera is more than 100 frames / second, and the frame rate of the ultra-high resolution camera is 10 frames / second. In other words, the frame rate of the other camera is more than 10 times the frame rate of the ultra-high resolution camera.

[0123] The central brain 15 associates the point information output from the MoPU 12 with the tag information output from the IPU 11. For example, due to the frame rate difference between the other cameras and the ultra-high resolution camera, the central brain 15 has a state where the point information related to the object is obtained but the tag information is not obtained. In this state, the central brain 15 recognizes the x-coordinate value and the y-coordinate value of the object based on the point information, but does not recognize what the object is.

[0124] Afterwards, when the tag information related to the above object is obtained, the central brain 15 derives the category of the tag information (for example: PERSON (human)). In addition, the central brain 15 associates the tag information with the point information obtained in the above. Thus, the central brain 15 recognizes the x-coordinate value and the y-coordinate value of the object based on the point information, and recognizes what the object is. The central brain 15 is an example of a "third processor".

[0125] Here, when there are multiple objects captured by the ultra-high resolution camera and other cameras, such as object A and object B, the central brain 15 associates the point information and label information related to each object in the following manner. Due to the difference in frame rate between the above-mentioned other cameras and the ultra-high resolution camera, the central brain 15 is in a state where it has obtained the point information related to object A and object B (hereinafter referred to as "point information A" and "point information B"), but has not obtained the label information. In this state, the central brain 15 recognizes the x-coordinate value and y-coordinate value of object A based on point information A, and recognizes the x-coordinate value and y-coordinate value of object B based on point information B, but does not recognize what these objects are.

[0126] Afterwards, when one tag information is obtained, the central brain 15 derives the category of the one tag information (for example: PERSON). And, the central brain 15 determines the point information associated with the one tag information based on the position information output from the IPU 11 together with the one tag information and the position information contained in the acquired point information A and point information B. For example, the central brain 15 determines the point information containing position information, which indicates the position closest to the position of the object indicated by the position information output from the IPU 11, and associates the point information with the one tag information. In the case where the point information determined in the above is point information A, the central brain 15 associates the one tag information with point information A, identifies the x-coordinate value and y-coordinate value of object A based on point information A, and identifies what object A is.

[0127] As described above, when there are multiple objects photographed by ultra-high-resolution cameras and other cameras, the central brain 15 associates point information with label information based on the position information output from the IPU 11 and the position information contained in the point information output from the MoPU 12.

[0128] In addition, the central brain 15 recognizes objects (people, animals, roads, signals, signs, crosswalks, obstacles, buildings, etc.) existing around the vehicle 100 based on the image and label information output from the IPU 11. In addition, the central brain 15 recognizes the position and movement of the recognized objects existing around the vehicle 100 based on the point information output from the MoPU 12. Based on the information obtained by the recognition, the central brain 15 controls the automatic driving of the vehicle 100 by, for example, controlling the motor driving the wheels (speed control), braking control, and steering wheel control. For example, the central brain 15 controls the automatic driving of the vehicle 100 based on the position information and motion information included in the point information output from the MoPU 12 to avoid collision with an object. In the central brain 15, the GNPU 13 can be responsible for processing related to image recognition, and the CPU 14 can be responsible for processing related to the control of the vehicle 100.

[0129] Typically, ultra-high-resolution cameras are used for image recognition in autonomous driving. Here, based on the image captured by the ultra-high-resolution camera, it is possible to identify what the object contained in the image is. However, in the Level 6 era of autonomous driving, this alone is not enough. In the Level 6 era, it is also necessary to identify the movement of objects with higher accuracy. By using MoPU12 to identify the movement of objects with higher accuracy, it is possible to perform avoidance actions such as the vehicle 100 traveling by autonomous driving to avoid obstacles with higher accuracy. However, with an ultra-high-resolution camera, only about 10 frames of images can be acquired per second, and the accuracy of analyzing the movement of objects is lower than that of a camera equipped with MoPU 12. On the other hand, with a camera equipped with MoPU 12, it is possible to output at a high frame rate of 100 frames per second, for example.

[0130] Therefore, the information processing device 10 involved in the second embodiment has two independent processors, namely, IPU 11 and MoPU 12. The information processing device 10 assigns the IPU 11 built in the ultra-high resolution camera the role of obtaining the information required to identify what the photographed object is, and assigns the MoPU 12 built in the other camera the role of detecting the position and movement of the object. MoPU 12 captures the photographed object as a point, and analyzes the coordinates of the point in which direction on at least the x-axis and y-axis in the above-mentioned three-dimensional orthogonal coordinate system and at what speed. Since the overall outline of the object and what the object is can be detected through the image from the ultra-high resolution camera, as long as the MoPU 12 knows how the center point of the object moves, it is known how the object as a whole behaves.

[0131] According to the method of analyzing only the movement and speed of the center point of the object, the amount of data output to the central brain 15 can be greatly suppressed compared to the case of determining how the entire image of the object moves, and the amount of calculation in the central brain 15 can be greatly reduced. For example, in the case of outputting a 1000 pixel × 1000 pixel image to the central brain 15 at a frame rate of 1000 frames / second, if color information is included, 4 billion bits / second of data will be output to the central brain 15. By outputting only the point information representing the movement of the center point of the object, the MoPU 12 can compress the amount of data output to the central brain 15 to 20,000 bits / second. That is, the amount of data output to the central brain 15 is compressed to one-twentieth of a million.

[0132] In this way, by combining and utilizing the low frame rate and high resolution image and label information output from the IPU 11 and the high frame rate and lightweight point information output from the MoPU 12, object recognition including the movement of the object can be achieved with a small amount of data.

[0133] Furthermore, in the information processing device 10 , by associating the point information output from the MoPU 12 with the tag information output from the IPU 11 by the central brain 15 , it is possible to grasp information on what kind of object is performing what kind of movement.

[0134] (Third Embodiment)

[0135] Next, a second embodiment according to the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0136] Fig.10 1 is a second block diagram showing an example of the structure of the information processing device 10. Fig.10 As shown, the information processing device 10 mounted on the vehicle 100 includes a MoPU 12L corresponding to a left eye, a MoPU 12R corresponding to a right eye, an IPU 11 , and a central brain 15 .

[0137] The MoPU 12L includes a camera 30L, a radar 32L, an infrared camera 34L, and a core 17L. In addition, the MoPU 12R includes a camera 30R, a radar 32R, an infrared camera 34R, and a core 17R. It should be noted that, hereinafter, when the MoPU 12L and the MoPU 12R are not distinguished, they are referred to as "MoPU 12", when the camera 30L and the camera 30R are not distinguished, they are referred to as "camera 30", when the radar 32L and the radar 32R are not distinguished, they are referred to as "radar 32", when the infrared camera 34L and the infrared camera 34R are not distinguished, they are referred to as "infrared camera 34", and when the core 17L and the core 17R are not distinguished, they are referred to as "core 17".

[0138] The camera 30 provided in the MoPU 12 captures an object at a frame rate (120, 240, 480, 960, or 1920 frames / second) greater than that of the ultra-high resolution camera (e.g., 10 frames / second) provided in the IPU 11. The frame rate of the camera 30 is variable. The camera 30 is an example of a "first camera".

[0139] The radar 32 included in the MoPU 12 acquires a radar signal based on a reflected wave of electromagnetic waves irradiated on an object and reflected from the object. The infrared camera 34 included in the MoPU 12 is a camera that captures infrared images.

[0140] The core 17 (e.g., composed of one or more CPUs) of the MoPU 12 extracts feature points from each frame of the image captured by the camera 30, and outputs the x-coordinate value and the y-coordinate value of the object in the three-dimensional orthogonal coordinate system as point information. The core 17 uses, for example, the center point (center of gravity) of the object extracted from the image as a feature point. It should be noted that the point information output by the core 17 is similar to that in the above-mentioned embodiment, and includes position information and motion information.

[0141] IPU 11 is equipped with an ultra-high resolution camera (not shown), and outputs an image of an object captured by the ultra-high resolution camera, label information indicating the category of the object, and position information indicating the position of the object in the camera coordinate system of the ultra-high resolution camera.

[0142] The central brain 15 obtains the point information output from the MoPU 12, and the image, label information, and position information output from the IPU 11. Furthermore, the central brain 15 associates the label information related to the object existing at the position corresponding to the position information contained in the point information output from the MoPU 12 and the position information output from the IPU 11 with the point information. Thus, in the information processing device 10, it is possible to associate the information about what the object represented by the label information is with the position and movement of the object represented by the point information.

[0143] Here, the MoPU 12 changes the frame rate of the camera 30 according to a prescribed factor. In the present embodiment, as an example of a prescribed factor, the MoPU 12 changes the frame rate of the camera 30 according to a score related to the external environment. In this case, the MoPU 12 calculates a score related to the external environment for the vehicle 100, and changes the frame rate of the camera 30 according to the calculated score. And, the MoPU 12 outputs a control signal for capturing an image at the changed frame rate to the camera 30. Thus, the camera 30 captures an image at the frame rate indicated by the control signal. With this structure, according to the information processing device 10, it is possible to capture an image of an object at a frame rate suitable for the external environment.

[0144] It should be noted that the information processing device 10 mounted on the vehicle 100 is equipped with a plurality of sensors not shown. The MoPU 12 calculates the degree of danger related to the movement of the vehicle 100 as a score related to the external environment for the vehicle 100 based on the sensor information obtained from the plurality of sensors (e.g., the center of gravity shift of the body weight, the detection of the material of the road, the detection of the temperature of the outside air, the detection of the humidity of the outside air, the detection of the lateral oblique inclination angle of the up and down direction of the slope, the freezing mode of the road, the detection of the amount of water, the material of each tire, the wear condition, the detection of the air pressure, the width of the road, the presence or absence of overtaking prohibition, the vehicle model information of the oncoming vehicle, the front and rear vehicles, the cruising state of these vehicles, or the surrounding conditions (birds, animals, football, accident vehicles, earthquakes, fires, wind, typhoons, heavy rain, light rain, blizzards, fog, etc.) and point information. The degree of danger indicates the degree of danger in which the vehicle 100 will travel in the future. In this case, the MoPU 12 changes the frame rate of the camera 30 according to the calculated degree of danger. The vehicle 100 is an example of a "moving object". With this configuration, the information processing device 10 can change the frame rate of the camera 30 according to the degree of danger associated with the movement of the vehicle 100. The sensor is an example of a “detection unit”, and the sensor information is an example of “detection information”.

[0145] For example, the higher the calculated danger level is, the higher the frame rate of the camera 30 is set by the MoPU 12. When the calculated danger level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames per second. In addition, when the calculated danger level is greater than or equal to the first threshold and less than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to any one of 240, 480, and 960 frames per second. In addition, when the calculated danger level is greater than or equal to the second threshold, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames per second. It should be noted that, in any of the above-mentioned danger levels, in addition to causing the camera 30 to capture images at the selected frame rate, the MoPU 12 can also output control signals to the radar 32 and the infrared camera 34 so that radar signals are acquired and infrared images are captured at values ​​corresponding to the frame rate.

[0146] For example, the lower the calculated danger level is, the lower the frame rate of the camera 30 is set by the MoPU 12. When the frame rate of the camera 30 is set to 1920 frames / second, if the calculated danger level is greater than or equal to the first threshold and less than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to any one of 240, 480, and 960 frames / second. In addition, when the frame rate of the camera 30 is set to 1920 frames / second, if the calculated danger level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / second. Furthermore, when the frame rate of the camera 30 is set to any one of 240, 480, and 960 frames / second, if the calculated danger level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / second. It should be noted that, in this case, similar to the above, control signals can be output to the radar 32 and the infrared camera 34 so that radar signals are acquired and infrared images are captured at values ​​corresponding to the frame rate of the changed camera 30 .

[0147] In addition, MoPU 12 can use long-tail event AI (Artificial Intelligence) DATA (for example, trip data of a vehicle equipped with Level 5 autonomous driving control method) or map information and other driving-related big data known before the vehicle 100 travels as information for predicting the degree of danger to calculate the degree of danger.

[0148] In the above, the degree of danger is calculated as a score related to the external environment, but the index of the score related to the external environment is not limited to the degree of danger. For example, the MoPU 12 can calculate a score related to the external environment different from the degree of danger based on the moving direction or speed of the object shown by the camera 30, and change the frame rate of the camera 30 according to the score. The following is a description of the following case: the MoPU 12 calculates a speed score, which is a score related to the speed of the object shown by the camera 30, and changes the frame rate of the camera 30 according to the speed score. As an example, the speed score is set so that the faster the speed of the object is, the higher the speed score is, and the slower the speed of the object is, the lower the speed score is. And, the higher the calculated speed score is, the higher the frame rate of the camera 30 is, and the lower the calculated speed score is, the lower the frame rate of the camera 30 is. Therefore, when the calculated speed score becomes equal to or greater than the threshold value due to the fast speed of the object, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames / second. In addition, when the calculated speed score becomes less than the threshold value due to the slow speed of the object, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / second. It should be noted that in this case, similar to the above, a control signal can be output to the radar 32 and the infrared camera 34 so that the radar signal is acquired and the infrared image is captured at a value corresponding to the changed frame rate of the camera 30.

[0149] Next, the following case is described: the MoPU 12 calculates a direction score, which is a score related to the moving direction of the object shown by the camera 30, and changes the frame rate of the camera 30 according to the direction score. As an example, the direction score is set so that the direction score becomes higher when the moving direction of the object is a direction approaching the road, and the direction score becomes lower when the moving direction of the object is a direction away from the road. And, the higher the calculated direction score, the higher the frame rate of the camera 30 is made by the MoPU 12, and the lower the calculated direction score, the lower the frame rate of the camera 30 is made by the MoPU 12. Specifically, the MoPU 12 determines the moving direction of the object by using AI or the like, and calculates the direction score based on the determined moving direction. And, when the calculated direction score becomes greater than or equal to the threshold value because the moving direction of the object is a direction approaching the road, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames / second. In addition, when the calculated direction score becomes less than the threshold value because the moving direction of the object is away from the road, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / second. It should be noted that in this case, similar to the above, a control signal can be output to the radar 32 and the infrared camera 34 so that the radar signal is acquired and the infrared image is captured at a value corresponding to the changed frame rate of the camera 30.

[0150] In addition, the MoPU 12 may output point information only for objects whose calculated scores related to the external environment are greater than or equal to a prescribed threshold value. In this case, for example, the MoPU 12 may determine whether to output point information related to the object based on the moving direction of the object reflected by the camera 30. For example, the MoPU 12 may not output point information related to objects that have little impact on the driving of the vehicle 100. Specifically, the MoPU 12 calculates the moving direction of the object reflected by the camera 30, and does not output point information related to objects such as pedestrians that are gradually moving away from the road. On the other hand, the MoPU 12 outputs point information related to objects approaching the road (for example, pedestrians and other objects that are about to rush out to the road). Through this structure, according to the information processing device 10, point information related to objects that have little impact on the driving of the vehicle 100 may not be output.

[0151] In addition, in the above description, the case where the MoPU 12 calculates the danger level is exemplified, but the disclosed technology is not limited to this method. For example, the central brain 15 may calculate the danger level instead of the MoPU 12. In this case, the central brain 15 calculates the danger level related to the movement of the vehicle 100 as a score related to the external environment for the vehicle 100 based on the sensor information obtained from the various sensors and the point information output from the MoPU 12. And, the central brain 15 outputs an instruction to change the frame rate of the camera 30 according to the calculated danger level to the MoPU 12.

[0152] In addition, in the above description, an example is given of a case where the MoPU 12 outputs point information based on an image captured by a camera, but the disclosed technology is not limited to this method. For example, the MoPU 12 may output point information based on a radar signal and an infrared image instead of based on an image captured by the camera 30. Similar to the image captured by the camera 30, the MoPU 12 can derive the x-coordinate value and the y-coordinate value of the object from the infrared image of the object captured by the infrared camera 34. The radar 32 can obtain three-dimensional point cloud data of the object based on the radar signal. In other words, the radar 32 can detect the coordinate of the z-axis in the above-mentioned three-dimensional orthogonal coordinate system. Here, the z-axis is an axis along the depth direction of the object and the direction of travel of the vehicle 100, and hereinafter, the coordinate value of the z-axis is recorded as a "z-coordinate value". In this case, the MoPU 12 uses the principle of a stereo camera to combine the x-coordinate value and the y-coordinate value of the object captured by the infrared camera 34 with the z-coordinate value of the object represented by the three-dimensional point cloud data at the same timing as the timing when the radar 32 acquires the three-dimensional point cloud data of the object, and derives the coordinate values ​​of the three coordinate axes (x-axis, y-axis, and z-axis) of the object as point information. And, the MoPU 12 outputs the derived point information to the central brain 15.

[0153] In addition, in the above description, the case where the MoPU 12 derives point information is exemplified, but the disclosed technology is not limited to this method. For example, the central brain 15 can derive point information instead of the MoPU 12. The central brain 15 derives point information by combining information detected by, for example, the camera 30L, the camera 30R, the radar 32, and the infrared camera 34. As a specific example, the central brain 15 performs three-point measurement based on the x-coordinate value and the y-coordinate value of the object captured by the camera 30L and the x-coordinate value and the y-coordinate value of the object captured by the camera 30R, thereby deriving the coordinate values ​​of the three coordinate axes (x-axis, y-axis, and z-axis) of the object as point information.

[0154] In addition, in the above description, an example is given of a case where the central brain 15 controls the automatic driving of the vehicle 100 based on the image and label information output from the IPU 11 and the point information output from the MoPU 12, but the disclosed technology is not limited to this method. For example, the central brain 15 can control the motion of the robot based on the above information output from the IPU 11 and the MoPU 12. The robot can be a humanoid intelligent robot that performs work instead of humans. In this case, the central brain 15 controls the motion of the robot's arms, palms, fingers, and feet based on the above information output from the IPU 11 and the MoPU 12, so that it can hold, grab, hold, carry, move, carry, throw, kick, and avoid objects. In the case where the central brain 15 controls the motion of the robot, the IPU 11 and the MoPU 12 can be mounted at the positions of the right eye and the left eye of the robot. That is, the IPU 11 and the MoPU 12 for the right eye can be mounted on the right eye, and the IPU 11 and the MoPU 12 for the left eye can be mounted on the left eye.

[0155] (Fourth Embodiment)

[0156] Next, a third embodiment according to the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0157] As an example, the information processing device 10 according to the fourth embodiment includes the same Fig. 9 The structure shown.

[0158] The MoPU 12 involved in the fourth embodiment outputs the coordinate values ​​of at least two diagonal points among the vertices of the polygon that surrounds the outline of the object recognized from the image captured by the other camera as point information. The coordinate values ​​are the x-coordinate values ​​and y-coordinate values ​​of the object in the above-mentioned three-dimensional orthogonal coordinate system, similar to the second embodiment.

[0159] Fig.11is an explanatory diagram showing an example of point information output by the MoPU 12. Fig.11 In the figure, for each of the four objects included in the image captured by the other cameras, the MoPU 12 shows a bounding box 21, a bounding box 22, a bounding box 23, and a bounding box 24 formed by surrounding the outline of the object with a quadrilateral. And, Fig.11 The example shows a method in which the MoPU 12 outputs the coordinate values ​​of two points as diagonal points in the vertices of the quadrilateral bounding boxes 21, 22, 23, and 24 that surround the outline of the object as point information. In this way, the MoPU 12 can capture the object not as a point but as an object of a certain size.

[0160] Furthermore, when capturing an object as an object of a certain size, the MoPU 12 may output the coordinate values ​​of the multiple vertices of the polygon surrounding the outline of the object as point information instead of the coordinate values ​​of the two diagonal points among the vertices of the polygon surrounding the outline of the object recognized from the image captured by the other camera. Fig.11 For example, MoPU 12 outputs the coordinate values ​​of all four vertices of bounding box 21, bounding box 22, bounding box 23, and bounding box 24 of the quadrilateral surrounding the outline of the object as point information.

[0161] (Fifth Embodiment)

[0162] Next, a fifth embodiment according to the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0163] As an example, the information processing device 10 according to the fifth embodiment includes the same Fig. 9 The structure shown.

[0164] The vehicle 100 equipped with the information processing device 10 according to the fifth embodiment includes a sensor composed of at least one of a radar, a LiDAR, a high-pixel / telephoto / ultra-wide-angle / 360-degree / high-performance camera, a visual sensor, a sound sensor, an ultrasonic sensor, a vibration sensor, an infrared sensor, an ultraviolet sensor, an electric wave sensor, a temperature sensor, and a humidity sensor. As sensor information obtained by the information processing device 10 from the sensor, there can be cited the shift of the center of gravity of the body weight, the detection of the material of the road, the detection of the temperature of the outside air, the detection of the humidity of the outside air, the detection of the lateral oblique inclination angle of the up and down direction of the slope, the freezing mode of the road, the detection of the moisture content, the material of each tire, the wear condition, the detection of the air pressure, the width of the road, the presence or absence of overtaking prohibition, the vehicle model information of the oncoming vehicle, the front and rear vehicles, the cruising state of these vehicles, or the surrounding conditions (birds, animals, football, accident vehicles, earthquakes, fires, wind, typhoons, heavy rain, light rain, blizzards, fog, etc.). The sensor is an example of a "detection unit", and the sensor information is an example of "detection information".

[0165] The central brain 15 involved in the fifth embodiment calculates control variables for controlling the automatic driving of the vehicle 100 based on the sensor information detected by the sensor. The central brain 15 obtains sensor information every nanosecond. Specifically, the central brain 15 calculates control variables for controlling the wheel speed, inclination and suspension supporting the wheels of each of the four wheels of the vehicle 100. It should be noted that the inclination of the wheel includes both the inclination of the wheel relative to the axis horizontal to the road and the inclination of the wheel relative to the axis perpendicular to the road. In this case, the central brain 15 calculates a total of 16 control variables, which are used to control the wheel speed of each of the four wheels, the inclination of each of the four wheels relative to the axis horizontal to the road, the inclination of each of the four wheels relative to the axis perpendicular to the road, and the suspension supporting the four wheels respectively.

[0166] Furthermore, the central brain 15 controls the automatic driving of the vehicle 100 based on the control variables calculated in the above, the point information output from the MoPU 12, and the tag information output from the IPU 11. Specifically, the central brain 15 controls the in-wheel motors mounted on the four wheels based on the above 16 control variables, thereby controlling the wheel speed, tilt, and suspensions supporting the four wheels of each of the four wheels of the vehicle 100 to perform automatic driving. In addition, the central brain 15 identifies the position and movement of the object identified as what exists around the vehicle 100 based on the point information and tag information, and controls the automatic driving of the vehicle 100 based on the information obtained by the identification to avoid, for example, collision with the object. In this way, the central brain 15 controls the automatic driving of the vehicle 100, so that, for example, when the vehicle 100 is driving on a mountain road, it can make the optimal turn that matches the mountain road, and when the vehicle 100 is parked in a parking lot, it can drive at the optimal angle that matches the parking lot.

[0167] Here, the central brain 15 may be a unit that can infer control variables based on the above sensor information and information that can be obtained from a server (not shown) via a network using machine learning, more specifically, deep learning. In other words, the central brain 15 may be composed of AI.

[0168] The central brain 15 is the computing power of the above sensor information and long-tail event AI data every nanosecond. By using the computing power used to achieve level 6 (hereinafter also referred to as "level 6 computing power"), the control variable can be obtained by performing multivariate analysis based on the integral method shown in the following formula (1) (for example, refer to formula (2)). More specifically, while the integral value of the Delta value of various ultra-high resolutions (Ultra High Resolution) can be obtained with the computing power of level 6, each control variable can be obtained at the edge level and in real time, and the result (that is, each control variable) that occurs in the next nanosecond can be obtained with the highest probability value. In order to achieve this, the integral value obtained by time-integrating the Delta value (for example, the change value of a small time) of the function (in other words, the function representing the change of each variable) that can determine each variable such as air resistance, road resistance, road element (for example, garbage) and slip coefficient (for example, the above sensor information and information that can be obtained via the network) is input into the deep learning model of the central brain 15 (for example, a learned model obtained by deep learning of a neural network). The deep learning model of the central brain 15 outputs a control variable corresponding to the input integral value (eg, a control variable with the highest confidence level (ie, evaluation value)). The control variable is output in nanoseconds.

[0169] [Calculation formula 1]

[0170]

[0171] [Calculation formula 2]

[0172] V n =DL(f(A,B,C,D,...,N)(dA n / dt)) (2)

[0173] It should be noted that, as an example, in formula (1), "f(A)" is a formula that simplifies a function that represents the change of various variables such as air resistance, road resistance, road elements (such as garbage) and sliding coefficient. In addition, as an example, formula (1) is a formula that represents the time integral v of "f(A)" from time a to time b. DL in formula (2) represents deep learning (for example, a deep learning model optimized by deep learning of a neural network), dAn / dt represents the Delta value of f(A, B, C, D, ..., N), A, B, C, D, ..., N represent various variables such as air resistance, road resistance, road elements (such as garbage) and sliding coefficient, f(A, B, C, D, ..., N) represents a function used to represent the change of A, B, C, D, ..., N, and Vn represents the value (control variable) output from the deep learning model optimized by deep learning of a neural network.

[0174] It should be noted that the example of inputting the integral value obtained by time-integrating the Delta value of the function to the deep learning model of the central brain 15 is listed here, but this is only an example. It can also be set, for example, that the deep learning model of the central brain 15 infers the integral value (for example, the result occurring in the next nanosecond) obtained by time-integrating the Delta value of the function representing the change of each variable such as air resistance, road resistance, road element and slip coefficient, and the central brain 15 obtains the integral value with the highest confidence (that is, evaluation value) as the inference result every nanosecond.

[0175] In addition, examples of inputting integral values ​​to or outputting integral values ​​from a deep learning model are listed here, but this is only an example, and the technology disclosed herein is also valid even if integral values ​​are not used. For example, at least one control variable may be inferred by a deep learning model optimized by deep learning a neural network using the following supervisory data, where the supervisory data uses values ​​equivalent to A, B, C, D, ..., N as example data and values ​​equivalent to at least one control variable (e.g., the result occurring in the next billionth of a second) as correct answer data.

[0176] The control variables obtained by the central brain 15 can be further refined by increasing the number of deep learning. For example, more accurate control variables can be calculated using massive data or long-tail event AI data such as the rotation of tires or motors, steering angles or road materials, weather, garbage or the impact of quadratic deceleration, slippage, steering or speed control methods for losing balance or regaining balance.

[0177] (Sixth Embodiment)

[0178] Next, a fifth embodiment according to the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0179] Fig.12 is a third block diagram showing an example of the structure of the information processing device 10. It should be noted that, Fig.12 Only a part of the configuration of the information processing device 10 is shown.

[0180] like Fig.12 As shown, through the MoPU 12, the visible light image and infrared image of the object captured by the camera 30 are input to the core 17 at a frame rate of 100 frames / second or more. The camera 30 is configured to include a visible light camera 30A capable of capturing a visible light image of the object and an infrared camera 30B capable of capturing an infrared image of the object. In addition, the core 17 outputs point information to the central brain 15 based on at least one of the input visible light image and infrared image.

[0181] Here, in the case where an object can be identified from a visible light image of the object captured by the visible light camera 30A, the core 17 outputs point information based on the visible light image. On the other hand, in the case where the object cannot be captured from the visible light image due to a prescribed factor, the core 17 outputs point information based on the infrared image of the object captured by the infrared camera 30B. For example, imagine a case where the core 17 cannot capture the object from the visible light image due to the influence of darkness as a prescribed factor. In this case, the core 17 detects the heat of the object using the infrared camera 30B, and outputs the point information of the object based on the infrared image as the detection result. It should be noted that, without limitation to this, the core 17 can also output point information based on the visible light image and the infrared image.

[0182] In addition, the MoPU 12 synchronizes the timing of capturing the visible light image through the visible light camera 30A with the timing of capturing the infrared image through the infrared camera 30B. Specifically, the MoPU 12 outputs a control signal to the camera 30 so that the visible light image and the infrared image are captured at the same timing. As a result, the number of images captured per second by the visible light camera 30A is synchronized with the number of images captured per second by the infrared camera 30B (for example, 1920 frames / second).

[0183] (Seventh Embodiment)

[0184] Next, a seventh embodiment according to the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0185] Fig.13 4 is a fourth block diagram showing an example of the structure of the information processing device 10. It should be noted that Fig.13 Only a part of the structure of the information processing device 10 is shown.

[0186] like Fig.13 As shown, through the MoPU 12, the image of the object captured by the camera 30 and the radar signal of the reflected wave reflected from the object based on the electromagnetic wave irradiated on the object by the radar 32 are respectively input to the core 17 at a frame rate of more than 100 frames / second. And, the core 17 outputs the point information to the central brain 15 based on the input image of the object and the radar signal. The core 17 can derive the x-coordinate value and y-coordinate value of the object from the input image of the object. As described above, the radar 32 can obtain the three-dimensional point cloud data of the object based on the radar signal and detect the coordinate of the z-axis in the above-mentioned three-dimensional orthogonal coordinate system. In this case, the core 17 uses the principle of a stereo camera to combine the x-coordinate value and the y-coordinate value of the object captured by the camera 30 at the same timing as the timing of the three-dimensional point cloud data of the object acquired by the radar 32, and the z-coordinate value of the object represented by the three-dimensional point cloud data, and derive the coordinate values ​​of the three coordinate axes (x-axis, y-axis and z-axis) of the object as point information. It should be noted that the image of the object input into the core 17 may also include at least one of a visible light image and an infrared image.

[0187] In addition, MoPU 12 synchronizes the timing of capturing images through camera 30 with the timing of acquiring three-dimensional point cloud data of an object based on radar signals by radar 32. Specifically, MoPU 12 captures images at the same timing and outputs control signals to camera 30 and radar 32 to acquire three-dimensional point cloud data of the object. Thus, the number of images captured per second by camera 30 is synchronized with the number of three-dimensional point cloud data acquired per second by radar 32 (for example, 1920 frames / second). In this way, the number of images captured per second by camera 30 and the number of three-dimensional point cloud data acquired per second by radar 32 are greater than the frame rate of the ultra-high resolution camera possessed by IPU11, that is, the number of images captured per second by the ultra-high resolution camera.

[0188] (Eighth Embodiment)

[0189] Next, an eighth embodiment of the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0190] As an example, the information processing device 10 according to the eighth embodiment includes the same Fig. 9 The structure shown.

[0191] The central brain 15 involved in the eighth embodiment associates the point information output from the MoPU 12 at the same timing as the timing of the IPU 11 outputting the tag information with the tag information. In addition, when the central brain 15 outputs new point information from the MoPU 12 after associating the point information with the tag information, the new point information is also associated with the tag information. The new point information is point information of the same object as the object represented by the point information after the association with the tag information, and is one or more point information from the time the association is made to the output of the next tag information. In the seventh embodiment, similar to the above-mentioned embodiment, the frame rate of other cameras with built-in MoPU 12 is greater than 100 frames / second (for example, 1920 frames / second), and the frame rate of the ultra-high resolution camera with built-in IPU 11 is 10 frames / second.

[0192] Fig.14 1 is an explanatory diagram showing an example of the association between point information and tag information. In the following description, the amount of point information output per second from the MoPU 12 is referred to as the "point information output rate", and the amount of tag information output per second from the IPU 11 is referred to as the "tag information output rate".

[0193] Fig.14The time series of the output rate of the point information P4 of the object B14 is shown. The output rate of the point information P4 related to the object B14 is 1920 frames / second. In addition, the point information P4 moves from right to left in the figure. The output rate of the tag information related to the object B14 is 10 frames per second, which is lower than the output rate of the point information P4.

[0194] First, at time t0, the tag information about the object B14 has not yet been output from the IPU 11. Therefore, at time t0, the central brain 15 recognizes the coordinate value (position information) of the object B14 based on the point information P4, but does not recognize what the object B14 is.

[0195] Next, at the time point of time t1, the tag information related to the object B14 is output from the IPU 11. Therefore, the central brain 15 derives the tag information "PERSON" for the object B14 based on the tag information. And the central brain 15 associates the tag information "PERSON" derived at the time point t1 with the coordinate value (position information) of the point information P4 output from the MoPU 12 at the time point t1. Thus, at the time point of time t1, the central brain 15 recognizes the coordinate value (position information) of the object B14 based on the point information P4, and recognizes what the object B14 is.

[0196] exist Fig.14 In the example, the timing at which the next tag information related to the object B14 is output from the IPU 11 is taken as the time t2. Therefore, at the time point of time t2, the central brain 15 derives the tag information "PERSON" for the object B14 based on the tag information output from the IPU 11. In addition, the central brain 15 associates the tag information "PERSON" derived at time t2 with the coordinate value (position information) of the point information P4 output from the MoPU 12 at time t2.

[0197] Here, due to the frame rate difference between the other cameras with built-in MoPU 12 and the ultra-high-resolution camera with built-in IPU 11, during the period from time t1 to time t2, point information P4 related to object B14 is acquired through the central brain 15, and on the other hand, tag information is not acquired. In this case, for the point information P4 acquired during the period from time t1 to time t2, the central brain 15 associates the point information P4 with the tag information "PERSON" associated with the upcoming time t1. Here, the point information P4 acquired by the central brain 15 during the period from time t1 to time t2 is an example of "new point information". Fig.21 In the example shown, since a plurality of point information P4 is output from the MoPU 12 during the period from time t1 to time t2, the central brain 15 acquires the plurality of point information P4. Fig.14 In the example shown, for any of the plurality of point information P4 acquired during the period from time t1 to time t2, the central brain 15 associates it with the tag information "PERSON" associated with the upcoming time t1. Fig.14 Unlike the example shown, when one point information P4 is output from the MoPU 12 during the period from time t1 to time t2, the central brain 15 associates the one point information P4 with the tag information "PERSON" associated at the upcoming time t1.

[0198] Here, as for the central brain 15, even if there is a period in which the type of the object being tracked by the tracking motion is uncertain, the point information of the object is continuously output at a high frame rate, so the risk of losing the coordinate value (position information) of the object is low. Therefore, when the point information and the tag information are associated once, the central brain 15 can presumptively assign the previous tag information to the point information acquired during the period until the next tag information is acquired.

[0199] (Ninth Embodiment)

[0200] Next, a ninth embodiment of the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0201] When the information processing device 10 that controls the automatic driving of the vehicle 100 performs advanced calculations, heat generation becomes a problem. Therefore, the ninth embodiment provides a vehicle 100 having a cooling function for the information processing device 10 .

[0202] Fig.15 1 is an explanatory diagram showing a schematic structure of the vehicle 100. Fig.15 As shown, the vehicle 100 is equipped with an information processing device 10 , a cooling execution device 110 , and a cooling unit 120 .

[0203] The information processing device 10 of the ninth embodiment is a device for controlling the automatic driving of the vehicle 100. As an example, the information processing device 10 has the same Fig. 9The structure shown. The cooling execution device 110 obtains the detection result of the object based on the information processing device 10, and based on the detection result, causes the cooling unit 120 to execute cooling of the information processing device 10. The cooling unit 120 uses at least one cooling mechanism such as an air cooling mechanism, a water cooling mechanism, and a liquid nitrogen cooling mechanism to cool the information processing device 10. In the following, the cooling object in the information processing device 10 is set to be the central brain 15 (specifically, the CPU 14 constituting the central brain 15) that controls the automatic driving of the vehicle 100 for explanation, but it is not limited to this.

[0204] The information processing device 10 and the cooling execution device 110 are communicatively connected via a network not shown. The network may be any one of a vehicle network, the Internet, a local area network (LAN), and a mobile communication network. The mobile communication network may be based on any one of a 5G (5th Generation) communication method, an LTE (Long Term Evolution) communication method, a 3G (3rd Generation) communication method, a 6G (6th Generation) communication method, and subsequent communication methods.

[0205] Fig.16 1 is a block diagram showing an example of the functional structure of the cooling execution device 110. Fig.16 As shown, the cooling execution device 110 includes an acquisition unit 112 , an execution unit 114 , and a prediction unit 116 as a functional structure.

[0206] The acquisition unit 112 acquires the detection result of the object by the information processing device 10. For example, the acquisition unit 112 acquires point information of the object output from the MoPU 12 as the detection result.

[0207] The execution unit 114 executes cooling of the central brain 15 based on the detection result of the object acquired by the acquisition unit 112. For example, when it is recognized that the object is moving based on the point information of the object output from the MoPU 12, the execution unit 114 starts cooling of the central brain 15 by the cooling unit 120.

[0208] It should be noted that the execution unit 114 is not limited to executing cooling of the central brain 15 based on the detection results of the object, but can also execute cooling of the central brain 15 based on the predicted results of the working status of the information processing device 10.

[0209] Here, the prediction unit 116 predicts the working condition of the information processing device 10, specifically, the working condition of the central brain 15, based on the detection result of the object acquired by the acquisition unit 112. For example, the prediction unit 116 acquires a learning model stored in a predetermined storage area. And, the prediction unit 116 predicts the working condition of the central brain 15 by inputting the point information of the object output from the MoPU 12 acquired by the acquisition unit 112 into the learning model. Here, the learning model outputs the condition and change amount of the computing power of the central brain 15 as the working condition. In addition, the prediction unit 116 can predict and output the temperature change of the information processing device 10, specifically, the temperature change of the central brain 15 together with the working condition. For example, the prediction unit 116 predicts the temperature change of the central brain 15 based on the number of point information of the object output from the MoPU 12 acquired by the acquisition unit 112. In this case, the prediction unit 116 predicts that the greater the number of point information, the greater the temperature change, and predicts that the smaller the number of point information, the smaller the temperature change.

[0210] In the above case, the execution unit 114 starts cooling the central brain 15 by the cooling unit 120 based on the prediction result of the working condition of the central brain 15 by the prediction unit 116. For example, when the state and change amount of the computing power of the central brain 15 predicted as the working condition exceed a predetermined threshold value, the execution unit 114 starts cooling by the cooling unit 120. In addition, when the temperature based on the temperature change of the central brain 15 predicted as the working condition exceeds a prescribed threshold value, the execution unit 114 starts cooling by the cooling unit 120.

[0211] In addition, the execution unit 114 may also use a cooling mechanism corresponding to the prediction result of the temperature change of the central brain 15 by the prediction unit 116 to perform cooling on the central brain 15. For example, the execution unit 114 may use more cooling mechanisms to cause the cooling unit 120 to perform cooling when the predicted temperature of the central brain 15 is higher. As a specific example, when it is predicted that the temperature of the central brain 15 exceeds a first threshold, the execution unit 114 uses one cooling mechanism to cause the cooling unit 120 to perform cooling. On the other hand, when it is predicted that the temperature of the central brain 15 exceeds a second threshold higher than the first threshold, the execution unit 114 uses multiple cooling mechanisms to cause the cooling unit 120 to perform cooling.

[0212] In addition, the execution unit 114 can use a stronger cooling mechanism to cool the central brain 15 when the predicted temperature of the central brain 15 is higher. For example, when it is predicted that the temperature of the central brain 15 exceeds a first threshold, the execution unit 114 uses an air cooling mechanism to cause the cooling unit 120 to perform cooling. In addition, when it is predicted that the temperature of the central brain 15 exceeds a second threshold higher than the first threshold, the execution unit 114 uses a water cooling mechanism to cause the cooling unit 120 to perform cooling. Further, when it is predicted that the temperature of the central brain 15 exceeds a third threshold higher than the second threshold, the execution unit 114 uses a liquid nitrogen cooling mechanism to cause the cooling unit 120 to perform cooling.

[0213] Further, the execution unit 114 may determine the cooling mechanism used for cooling based on the number of point information of the object output from the MoPU 12 acquired by the acquisition unit 112. In this case, the execution unit 114 may perform cooling of the central brain 15 using a cooling mechanism that is more powerful as the number of point information increases. For example, when the number of point information exceeds the first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using an air cooling mechanism. In addition, when the number of point information exceeds a second threshold higher than the first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using a water cooling mechanism. Further, when the number of point information exceeds a third threshold higher than the second threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using a liquid nitrogen cooling mechanism.

[0214] As an opportunity for the central brain 15 to work, a moving object in the lane is sometimes detected. For example, when a moving object is detected in the lane while the vehicle 100 is performing autonomous driving, the central brain 15 sometimes performs computational processing for the object to control the vehicle 100. As described above, the heat generated when the central brain 15 that controls the autonomous driving of the vehicle 100 performs high-level computational processing becomes a problem. Therefore, the cooling execution device 110 involved in the eighth embodiment predicts the heat dissipation of the central brain 15 based on the detection result of the object by the information processing device 10, and performs cooling of the central brain 15 before or at the same time as the heat dissipation starts. Thereby, the central brain 15 is prevented from becoming hot during the autonomous driving of the vehicle 100, and high-level computations of the central brain 15 in the autonomous driving become possible.

[0215] As described above, the cooling execution device 110 involved in the ninth embodiment executes cooling of the central brain 15 based on the detection result of the object by the information processing device 10 (MoPU12). And when a predetermined condition is met, the cooling execution device 110 stops cooling the central brain 15. Specifically, when a predetermined condition is met, the execution unit 114 of the cooling execution device 110 stops the cooling of the central brain 15 by the cooling unit 120. In this way, by stopping the cooling of the central brain 15 when a predetermined condition is met, excessive cooling can be suppressed. As a result, the power consumption of the vehicle 100 can be reduced.

[0216] In addition, the predetermined condition may include, for example, a condition that the object detected by the information processing device 10 is no longer detected, or a condition that the object detected by the information processing device 10 is moving outside the detection range. That is, when the condition that the object detected by the information processing device 10 is no longer detected is satisfied, the execution unit 114 may stop cooling the central brain 15, and when the condition that the object detected by the information processing device 10 is moving outside the detection range is satisfied, the execution unit 114 may stop cooling the central brain 15. It should be noted that the state in which the object detected by the information processing device 10 is no longer detected may include, for example, a state in which the detected object is outside the detection range of the MoPU 12 (outside the detection range), that is, outside the photographic range of other cameras, or a state in which the object is stationary. In addition, the state in which the object detected by the information processing device 10 is moving outside the detection range may include a state in which the detected object is moving (relatively moving) outside the detection range of the MoPU 12, that is, outside the photographic range of other cameras. In this way, when the condition that the object detected by the information processing device 10 is no longer detected, or when the condition that the object detected by the information processing device 10 is tending to be out of the detection range is met, the cooling of the central brain 15 is stopped, thereby effectively suppressing excessive cooling of the central brain 15 (in other words, unnecessary cooling).

[0217] In addition, the execution unit 114 can determine whether to stop cooling the central brain 15 based on the relationship between the predetermined sight line in other cameras of the built-in MoPU 12 and the object detected by the information processing device 10. For example, when the object detected by the information processing device 10 is approaching the sight line, the execution unit 114 can maintain the cooling of the central brain 15, and when the detected object is away from the sight line, the execution unit 114 can stop cooling the central brain 15. In addition, when the positional relationship between the object detected by the information processing device 10 and the sight line does not change, the execution unit 114 can also consider the movement of the detected object in the z-axis direction (the front and rear direction of the vehicle) to determine whether to maintain or stop cooling the central brain 15.

[0218] In addition, when a predetermined condition is met, the execution unit 114 can stop cooling the central brain 15 after a predetermined time has passed. Specifically, when the condition that the object detected by the information processing device 10 is no longer detected is met, or when the condition that the object detected by the information processing device 10 is moving out of the detection range is met, the execution unit 114 can stop cooling the central brain 15 after a predetermined time has passed. Here, the predetermined time can be determined based on the type, size, acceleration, distance, etc. of the detected object. In this way, by stopping the cooling of the central brain 15 after a predetermined time has passed when a predetermined condition is met, the central brain 15 can be cooled during the period from when the predetermined condition is met to when the predetermined time has passed.

[0219] In addition, the prediction unit 116 predicts the working condition of the central brain 15 by inputting the point information of the object output from the MoPU 12 acquired by the acquisition unit 112 into the learning model, but can further predict the period when the temperature of the central brain 15 is below the threshold. The learning of the period when the temperature of the central brain 15 is below the threshold can be performed by using the data collected by the vehicle 100. For example, the cloud server connected to the central brain 15 can collect data from the vehicle 100 and perform learning. Specifically, for the input of the type, size, acceleration, distance, etc. of the object detected in the MoPU 12, supervision data with the assumed temperature of the central brain 15 as the output is prepared, and based on the supervision data, the cloud server generates a cooling model that has been learned. And, in the cooling execution device 110 into which the learned cooling model has been introduced, the predicted temperature of the central brain 15 is obtained by inputting the detection condition of the central brain 15 into the cooling model by the prediction unit 116, so that appropriate cooling can be performed on the central brain 15. It should be noted that, instead of outputting the expected temperature of the central brain 15 through the cooling model, the cooling conditions of the central brain 15 (for example, cooling intensity, cooling time, etc.) may be output. In this case, when learning the cooling model, learning using supervision data based on the cooling conditions is performed. It should be noted that the subject that performs learning is not limited to the cloud server, but may also be other devices. Here, the execution unit 114 may also stop cooling the central brain 15 when the predetermined condition is met, that is, until the temperature of the central brain 15 predicted by the prediction unit 116 is below the threshold (until). In this way, by stopping the cooling of the central brain 15 when the condition of the period predicted by the prediction unit 116 is met, overcooling of the central brain 15 can be effectively suppressed.

[0220] [Modifications]

[0221] In addition, the cooling execution device 110 can execute cooling of the central brain 15 based on the prediction result of the working condition of the central brain 15 by the prediction unit 116, and adjust the cooling amount of the central brain 15 based on the prediction result of the temperature change of the central brain 15 by the prediction unit 116. The prediction of the temperature change by the prediction unit 116 can be obtained as a temperature curve of the central brain 15 by inputting the prediction result of the working condition of the central brain 15 into the learned model for temperature prediction. The learned model for temperature prediction is a model that has been machine-learned in advance using various actual working conditions in the central brain 15 as input and the actual temperature for the working condition as output. As the "various working conditions" in the central brain 15, there are the proximity of the object to the vehicle 100, the approach number, the approach speed, etc. That is, the execution unit 114 of the cooling execution device 110 increases the cooling amount of the cooling mechanism when the prediction unit 116 predicts that the temperature change of the central brain 15 is a temperature rise, reduces the cooling amount of the cooling mechanism when the prediction result of the temperature change is a temperature drop, and maintains the cooling amount of the cooling mechanism when the prediction result of the temperature change is no temperature change. For example, when the state and change amount of the power calculation force of the central brain 15 predicted as the working condition exceed a predetermined threshold, and when the temperature rises or the temperature rise exceeds the predetermined threshold, the execution unit 114 increases the cooling amount to the central brain 15. Here, the "predetermined threshold" can also be set to a predetermined ratio relative to the peak value of the heat generation of the central brain 15, such as a value of 80%. In addition, the "predetermined threshold" can also change according to the external temperature, humidity, and degree of driving wind.

[0222] In addition, when the execution unit 114 uses an air cooling mechanism, the cooling amount of the central brain 15 is equivalent to the air volume. In addition, when the execution unit 114 uses a water cooling mechanism or a liquid nitrogen cooling mechanism, the cooling amount of the central brain 15 is equivalent to the temperature and circulation speed of the refrigerant.

[0223] In this way, the execution unit 114 executes cooling of the central brain 15 based on the prediction result of the prediction unit 116 on the working condition of the central brain 15, and the execution unit 114 adjusts the cooling amount of the central brain 15 based on the prediction result of the prediction unit 116 on the temperature change of the central brain 15, thereby suppressing overcooling and insufficient cooling of the central brain 15. In other words, the cooling amount from the execution unit 114 can be made close to an appropriate amount.

[0224] In addition, as for the cooling execution device 110, the prediction unit 116 can further predict the peak value of the temperature in the central brain 15, and the execution unit 114 can execute the cooling of the central brain 15 with the cooling amount corresponding to the peak value of the temperature predicted by the prediction unit 116. It should be noted that the cooling amount of the central brain 15 is pre-set according to the peak value of the temperature. In this way, since the execution unit 114 cools the central brain 15 with the cooling amount corresponding to the peak value of the temperature predicted by the prediction unit 116, it is possible to suppress overcooling and insufficient cooling of the central brain 15. That is, the cooling amount from the execution unit 114 can be closer to the appropriate amount.

[0225] Furthermore, in the cooling execution device 110, the execution unit 114 may execute cooling of the central brain 15 in a manner that the cooling amount corresponds to the peak value of the temperature before the peak value of the temperature predicted by the prediction unit 116 is reached. The trigger before the peak value of the temperature can be set to a case where the predicted temperature of the central brain 15 by the prediction unit 116 exceeds a predetermined set temperature, or a case where the predetermined time for the temperature of the central brain 15 to reach the peak value. In these cases, since the cooling amount of the central brain 15 is the cooling amount corresponding to the peak value of the temperature at the early stage of the temperature rise of the central brain 15, the temperature rise of the central brain 15 can be effectively suppressed. That is, efficient cooling of the central brain 15 can be achieved.

[0226] In addition, when the object is moving towards the vehicle 100 as a moving body in the detection result of the information processing device 10 (MoPU 12), the prediction unit 116 of the cooling execution device 110 can predict that the temperature of the central brain 15 will rise. In this way, when the prediction unit 116 predicts that the temperature of the central brain 15 will rise, the execution unit 114 increases the cooling amount of the central brain 15. Here, when the object detected by the information processing device 10 approaches the vehicle, it is more necessary to monitor the moving object, so the amount of calculation of the central brain 15 increases. Therefore, it is expected that the temperature of the central brain 15 will rise. Therefore, by increasing the cooling amount of the central brain 15 cooled by the execution unit, insufficient cooling of the central brain 15 can be suppressed. On the other hand, when the object detected by the information processing device 10 is away from the vehicle 100, the amount of calculation of the central brain 15 is reduced by excluding the object from the monitoring object. Therefore, it is expected that the temperature of the central brain 15 will decrease. Therefore, by reducing the cooling amount of the central brain 15 cooled by the execution unit 114, excessive cooling can be suppressed. In this way, when the detected object approaches or moves away from the vehicle 100, the central brain 15 can be efficiently cooled by adjusting the cooling amount of the execution unit 114 for cooling the central brain 15.

[0227] (Tenth Embodiment)

[0228] Next, a tenth embodiment according to the present embodiment will be described while omitting or simplifying the overlapping parts with the above-mentioned embodiment.

[0229] The MoPU 12 included in the information processing device according to the tenth embodiment derives the z coordinate value of the object as point information from the image of the object captured by the camera 30. Hereinafter, each aspect of the information processing device 10 according to the ninth embodiment will be described in sequence.

[0230] The information processing device 10 according to the first embodiment includes the same Fig.10 The structure shown.

[0231] In the first mode described above, the MoPU 12 derives the z coordinate value of the object as point information from the image of the object captured by the plurality of cameras 30, specifically, from the camera 30L and the camera 30R. As described above, when one MoPU 12 is used, the x coordinate value and the y coordinate value of the object can be derived as point information. Here, when two MoPUs 12 are used, the z coordinate value of the object can be derived as point information based on the image of the object captured by the two cameras 30 using the principle of a stereo camera. Therefore, the first mode derives the z coordinate value of the object as point information based on the image of the object captured by the camera 30L of the MoPU 12L and the camera 30R of the MoPU 12R, respectively, using the principle of a stereo camera.

[0232] The information processing device 10 according to the second embodiment includes the same Fig.17 The structure shown.

[0233] In the second mode described above, MoPU 12 derives the x-coordinate value, y-coordinate value, and z-coordinate value of the object as point information based on the image of the object captured by the camera 30 and the radar signal of the reflected wave reflected from the object by the electromagnetic wave irradiated on the object by the radar 32. As described above, the radar 32 is capable of obtaining three-dimensional point cloud data of the object based on the radar signal. That is, the radar 32 is capable of detecting the coordinates of the z-axis in the three-dimensional orthogonal coordinate system described above. In this case, MoPU 12 utilizes the principle of a stereo camera to combine the x-coordinate value and the y-coordinate value of the object captured by the camera 30 at the same timing as the timing of the three-dimensional point cloud data of the object acquired by the radar 32 with the z-coordinate value of the object represented by the three-dimensional point cloud data, and derives the coordinate values ​​of the three coordinate axes of the object as point information.

[0234] The information processing device 10 according to the third embodiment includes: Fig.17 The structure shown. Fig.17is a fifth block diagram showing an example of the structure of the information processing device 10. It should be noted that, Fig.17 Only a part of the configuration of the information processing device 10 is shown.

[0235] In the third mode described above, the MoPU 12 derives the z coordinate value of the object as point information based on the image of the object captured by the camera 30 and the result of capturing the structured light irradiated onto the object by the irradiation device 130 .

[0236] like Fig.17 As shown, through the MoPU 12, the image of the object captured by the camera 30 and the distortion information indicating the distortion of the pattern of the structured light, which is the result of capturing the structured light irradiated on the object by the irradiation device 130 by the camera 140, are input to the core 17 at a frame rate of more than 100 frames per second. And, the core 17 outputs the point information to the central brain 15 based on the input image of the object and the distortion information.

[0237] Here, as one of the methods for recognizing the three-dimensional position or shape of an object, there is a structured light method. The structured light method is to illuminate the object with structured light after patterning into dots, and obtain depth information based on the distortion of the pattern. The structured light method is disclosed in the reference document (http: / / ex-press.jp / wp-content / uploads / 2018 / 10 / 018_teledyne_3rd.pdf), for example.

[0238] Fig.17 The irradiation device 130 shown irradiates structured light onto the object. In addition, the camera 140 captures the structured light irradiated onto the object by the irradiation device 130. Furthermore, the camera 140 outputs distortion information based on the distortion of the captured pattern of the structured light to the core 17.

[0239] Here, the MoPU 12 synchronizes the timing of capturing an image through the camera 30 with the timing of capturing structured light through the camera 140. Specifically, the MoPU 12 outputs a control signal to the camera 30 and the camera 140 so that the camera 30 and the camera 140 capture images at the same timing. As a result, the number of images captured per second by the camera 30 is synchronized with the number of images captured per second by the camera 140 (for example, 1920 frames / second). In this way, the number of images captured per second by the camera 30 and the number of images captured per second by the camera 140 are greater than the frame rate of the ultra-high resolution camera possessed by the IPU 11, that is, the number of images captured per second by the ultra-high resolution camera.

[0240] Furthermore, the core 17 combines the x-coordinate value and the y-coordinate value of the object captured by the camera 30 at the same timing as the timing of capturing the structured light by the camera 140 with the distortion information based on the distortion of the pattern of the structured light, and derives the z-coordinate value of the object as point information.

[0241] The information processing device 10 according to the fourth aspect includes: Fig.18 The structure shown. Fig.18 6 is a block diagram showing an example of the structure of the information processing device 10. It should be noted that Fig.18 Only a part of the configuration of the information processing device 10 is shown.

[0242] Fig.18 The block diagram shown is in Fig. 9 A block diagram in which a lidar sensor 18 is added to the structure of the block diagram shown. The lidar sensor 18 is a sensor that acquires point cloud data including objects existing in a three-dimensional space and a road surface on which the vehicle 100 is traveling. The information processing device 10 can derive position information of the object in the depth direction, that is, the z coordinate value of the object, by using the point cloud data acquired by the lidar sensor 18. It should be noted that it is assumed that the point cloud data acquired by the lidar sensor 18 is acquired at intervals longer than the x coordinate values ​​and y coordinate values ​​of the object output from the MoPU 12. In addition, the MoPU 12 is equipped with a camera 30 similar to the above-mentioned aspects of the ninth embodiment.

[0243] In the fourth method, MoPU 12 uses the principle of a stereo camera to combine the x-coordinate value and the y-coordinate value of the object captured by the camera 30 at the same timing as the point cloud data of the object acquired by the Lidar sensor 18 with the z-coordinate value of the object represented by the point cloud data, and derives the coordinate values ​​of the three coordinate axes of the object as point information.

[0244] Here, in the fourth mode, the MoPU 12 derives the z coordinate value of the object at time t+1 as point information based on the x coordinate value, y coordinate value, and z coordinate value of the object at time t and the x coordinate value and y coordinate value of the object at the next time point (e.g., time t+1) of time t. Time t is an example of a "first time point", and time t+1 is an example of a "second time point". In the fourth mode, the z coordinate value of the object at time t+1 is derived using shape information, that is, a geometric shape. The details are described below.

[0245] Fig.19 is a diagram schematically showing the coordinate detection of an object in a time series. Fig.19 In , J represents the position of the object represented by the rectangle, and the position of the object moves from J1 to J2 in time series. Fig.19In the figure, the coordinate value of the object at time t when the object is located at J1 is (x1, y1, z1), and the coordinate value of the object at time t+1 when the object is located at J2 is (x2, y2, z2).

[0246] First, time t will be described.

[0247] The MoPU 12 derives the x-coordinate value and the y-coordinate value of the object from the image of the object captured by the camera 30. Next, the MoPU 12 integrates the z-coordinate value of the object represented by the point cloud data acquired from the Lidar sensor 18 with the x-coordinate value and the y-coordinate value, and derives the three-dimensional coordinate value (x1, y1, z1) of the object at time t.

[0248] Next, time t+1 will be described.

[0249] The MoPU 12 derives the z coordinate value of the object at time t + 1 based on the geometry of the space and the changes in the x coordinate value and the y coordinate value of the object from time t to time t + 1. The geometry of the space includes the shape of the road surface and the shape of the vehicle 100 obtained from the image captured by the ultra-high resolution camera of the IPU 11 and the point cloud data of the Lidar sensor.

[0250] The geometric shape representing the shape of the road surface is pre-generated at time t. The MoPU 12 combines the geometric shape representing the shape of the vehicle 100 with the geometric shape representing the shape of the road surface to simulate the vehicle 100 traveling on the road surface and estimate the movement of each axis of the x-axis, y-axis, and z-axis.

[0251] Therefore, the MoPU 12 derives the x-coordinate value and y-coordinate value of the object at time t+1 from the image of the object captured by the camera 30. The MoPU 12 simulates and calculates the movement of the z-axis when the x-coordinate value and y-coordinate value (x1, y1) of the object at time t changes to the x-coordinate value and y-coordinate value (x2, y2) at time t+1, thereby being able to derive the z-coordinate value of the object at time t+1. Furthermore, the MoPU 12 integrates the above x-coordinate value and y-coordinate value with the z-coordinate value to derive the three-dimensional coordinate value (x2, y2, z2) of the object at time t+1.

[0252] like Fig.19As shown, since the object moves in the depth direction together with the movement of the plane coordinates (i.e., the x-axis and the y-axis), in order to control the automatic driving of the vehicle 100 with high precision, it is also necessary to detect the movement in the z-axis direction. Here, the MoPU 12 is sometimes unable to obtain the z-coordinate value of the object as high-speed as the x-coordinate value and the y-coordinate value of the object that can be derived from the point cloud data of the Lidar sensor 18. Therefore, in the above-mentioned fourth method, it is assumed that the MoPU 12 derives the z-coordinate value of the object at time t+1 based on the x-coordinate value, y-coordinate value and z-coordinate value of the object at time t+1 and the x-coordinate value and y-coordinate value of the object at time t+1. Therefore, according to the information processing device 10 involved in the above-mentioned fourth method, through the MoPU 12, two-dimensional motion detection and three-dimensional motion detection based on high-speed frame shooting can be realized with high performance and low data usage.

[0253] In addition, in the above description, an example is given of a case where the MoPU 12 derives the z coordinate value of the object as point information from the image of the object captured by the camera 30, but the disclosed technology is not limited to this method. For example, the central brain 15 may derive the z coordinate value of the object as point information instead of the MoPU 12. In this case, the central brain 15 derives the z coordinate value of the object as point information by performing the processing performed by the MoPU 12 in the above description on the image of the object captured by the camera 30. As an example, the central brain 15 derives the z coordinate value of the object as point information from the image of the object captured by a plurality of cameras 30, specifically, by the camera 30L and the camera 30R. In this case, the central brain 15 utilizes the principle of stereo cameras and derives the z coordinate value of the object as point information based on the images of the object captured by the camera 30L of the MoPU 12L and the camera 30R of the MoPU 12R, respectively.

[0254] (Eleventh Embodiment)

[0255] Next, an eleventh embodiment according to the present embodiment will be described while omitting or simplifying portions overlapping with the above-described embodiment.

[0256] Fig. 20 7 is a seventh block diagram showing an example of the structure of the information processing device 10. It should be noted that, Fig. 20 Only a part of the configuration of the information processing device 10 is shown.

[0257] like Fig. 20As shown, the image of the object captured by the event camera 30C (hereinafter sometimes referred to as "event image") is input to the core 17 through the MoPU 12. In addition, the core 17 outputs the point information to the central brain 15 based on the input event image. It should be noted that the event camera is disclosed in the reference document (https: / / dendenblog.xyz / event-based-camera / ), for example.

[0258] Fig.21 It is an explanatory diagram for explaining an image of an object captured by the event camera 30C (event image). Fig.21 (A) is a diagram showing an object that is a photographic target by the event camera 30C. Fig.21 (B) is a diagram showing an example of an event image. Fig.21 (C) is a diagram showing an example of calculating the centroid of the difference between the image captured at the current moment and the image captured at the previous moment represented by the event image as point information. In the event image, the difference between the image captured at the current moment and the image captured at the previous moment is extracted as a point. Therefore, when the event camera 30C is used, for example, Fig.21 As shown in (B), Fig.21 Points of each moving part in the human figure region shown in (A) are extracted.

[0259] In contrast, Fig.21 As shown in (C), after extracting the person as the object, the core 17 extracts the coordinates of the feature points representing the person area (for example, only one point). This can suppress the amount of data transmitted to the central brain 15 and the memory 16. The event image can extract the person as the object at any frame rate, so in the case of the event camera 30C, in the above embodiment, it can also be extracted at a frame rate higher than the maximum frame rate of the camera 30 mounted on the MoPU 12 (for example: 1920 frames / second), and the point information of the object can be captured with high precision.

[0260] It should be noted that, similar to the above-mentioned embodiment, in the information processing device 10 according to the tenth embodiment, in addition to the event camera 30C, the MoPU 12 may also include a visible light camera 30A. In this case, in the MoPU 12, the visible light image and the event image of the object captured by the visible light camera 30A are respectively input to the core 17. And the core 17 outputs point information to the central brain 15 based on at least one of the input visible light image and the event image.

[0261] For example, when the core 17 is able to identify an object from a visible light image of the object captured by the visible light camera 30A, the core 17 outputs point information based on the visible light image. On the other hand, when the object cannot be captured from the visible light image due to a specified factor, the core 17 outputs point information based on the event image. The specified factor includes at least one of a case where the moving speed of the object is greater than a specified value and a case where the amount of light of the ambient light per unit time changes greater than a specified value. For example, when the object moves at a high speed and cannot be captured from the visible light image, the core 17 identifies the object based on the event image and outputs the x-coordinate value and y-coordinate value of the object as point information. In addition, when the object cannot be captured from the visible light image due to a sudden change in the amount of light of the ambient light such as backlight, the core 17 identifies the object based on the event image and outputs the x-coordinate value and y-coordinate value of the object as point information. Through this structure, according to the information processing device 10, it is possible to use the camera 30 for capturing the object according to the specified factors.

[0262] Fig. 22 An example of the hardware structure of a computer 1200 that functions as a management server 101, SoCBox 400, cooling execution device 500, information processing device 10, or cooling execution device 110 is schematically shown. The program installed in the computer 1200 can make the computer 1200 function as one or more "parts" of the device involved in this embodiment, or make the computer 1200 perform operations associated with the device involved in this embodiment or the one or more "parts", and / or can make the computer 1200 perform the process involved in this embodiment or the stage of the process. Such a program can be executed by the CPU 1212 to make the computer 1200 perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described in this specification.

[0263] The computer 1200 according to the above-mentioned embodiments includes a CPU 1212, a RAM 1214, and a graphic controller 1216 that can be connected to each other through a host controller 1210. The computer 1200 also includes a communication interface 1222, a storage device 1224, an input / output unit such as a DVD drive and an IC card drive, which are connected to the host controller 1210 via the input / output controller 1220. The DVD drive is a DVD-ROM drive and a DVD-RAM drive, etc. The storage device 1224 is a hard disk drive and a solid state drive, etc. The computer 1200 may also include a ROM 1230 and a conventional input / output unit such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0264] The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or in itself, and causes the image data to be displayed on the display device 1218.

[0265] The communication interface 1222 communicates with other electronic devices via a network.

[0266] The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs and data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0267] The ROM 1230 stores therein a boot program or the like executed by the computer 1200 at startup, and / or a program depending on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0268] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which is also an example of a computer-readable storage medium, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200, and the program and the aforementioned various types of hardware resources are made to cooperate. The device or method can be configured by realizing the operation or processing of information according to the use of the computer 1200.

[0269] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 can execute a communication program loaded into the RAM 1214, and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads the transmission data stored in the transmission buffer provided in the RAM 1214, the storage device 1224, a recording medium such as a DVD-ROM or an IC card, and transmits the read transmission data to the network, or writes the reception data received from the network to the reception buffer provided on the recording medium, etc.

[0270] In addition, the CPU 1212 can cause all or a necessary part of a file or a database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Next, the CPU 1212 can write the processed data back to the external recording medium.

[0271] Various types of information such as various types of programs, data, tables, and databases can be stored in the recording medium to receive information processing. The CPU 1212 can perform various types of processing on the data read from the RAM 1214, and write the results back to the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgments, conditional branches, unconditional branches, information retrieval / replacement, etc. recorded in various places of the present disclosure and specified by the instruction sequence of the program. In addition, the CPU 1212 can retrieve information in files, databases, etc. in the recording medium. For example, in the case where multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 can retrieve an entry that is consistent with the condition specifying the attribute value of the first attribute from the multiple entries, and read the attribute value of the second attribute stored in the entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that meets the predetermined condition.

[0272] The program or software module described above may be stored in a computer-readable storage medium on or near the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a computer-readable storage medium, thereby enabling the program to be provided to the computer 1200 via the network.

[0273] The flowcharts and boxes in the block diagrams in this embodiment may represent the stages of the process of performing an operation or the "parts" of the device having the function of performing an operation. Specific stages and "parts" may be implemented by dedicated circuits, programmable circuits supplied together with computer-readable instructions stored on a computer-readable storage medium, and / or processors supplied together with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuits may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, triggers, registers, and storage elements.

[0274] Computer-readable storage media may include any tangible device capable of storing instructions executed by an appropriate device, with the result that a computer-readable storage medium having instructions stored in a tangible device has a product including instructions that can be executed to generate a unit for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of the computer-readable storage medium may include a floppy disk (registered trademark) disk, a magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a Blu-ray disc (Blu-ray (registered trademark) Disk), a memory stick, an integrated circuit card, and the like.

[0275] Computer readable instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or any source code or object code described in any combination of one or more programming languages, wherein the one or more programming languages ​​include object-oriented programming languages ​​such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc. and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0276] The computer-readable instructions can be provided to a processor or programmable circuit of a general-purpose computer, a special-purpose computer or other programmable data processing device locally or through a local LAN (Local Area Network, LAN), a wide area network (Wide Area Network, WAN) such as the Internet, etc., so that the processor or programmable circuit of the general-purpose computer, special-purpose computer or other programmable data processing device executes the computer-readable instructions to generate a unit for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0277] The above embodiments are used to illustrate the technology of the present invention, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It should be clear to those skilled in the art that various changes or improvements can be made to the above embodiments. It can be seen from the description in the claims that the embodiments with such changes or improvements can also be included in the technical scope of the present invention.

[0278] It should be noted that the execution order of each process such as actions, sequences, steps and stages in the devices, systems, programs and methods shown in the claims, specifications and drawings is not specifically indicated as "before", "earlier than", etc., or as long as the output of the previous process is not used in the subsequent process, it can be implemented in any order. Even if the action flow in the claims, specifications and drawings is described using "first", "next", etc. for convenience, it does not mean that it must be implemented in this order.

[0279] In the above embodiment, the processing set to be executed by each processor (e.g., IPU 11, MoPU 12, and central brain 15) is merely exemplary, and the processor that executes each processing is not limited. For example, the processing set to be executed by MoPU 12 in the above embodiment can be executed by central brain 15 instead of MoPU 12, or by other processors other than IPU 11, MoPU 12, and central brain 15.

[0280] <Additional Note 1> (1)

[0282] An information processing device, wherein the information processing device comprises:

[0283] a first processor, which outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera; and

[0284] A second processor is configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera. (2)

[0286] The information processing device according to (1), wherein the information processing device includes a third processor,

[0287] The third processor associates the point information output from the first processor with the identification information output from the second processor. (3)

[0289] The information processing device according to (1) or (2), wherein:

[0290] The frame rate of the first camera is variable.

[0291] The first processor changes the frame rate of the first camera according to a specified factor. (4)

[0293] The information processing device according to (3), wherein:

[0294] The first processor calculates a score related to the external environment for a predetermined object. (5)

[0296] The information processing device according to (4), wherein:

[0297] The first processor changes the frame rate of the first camera according to the calculated score related to the external environment. (6)

[0299] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0300] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0301] Based on an image of the object captured by a second camera facing a direction corresponding to the first camera, identification information obtained by identifying the captured object is output. (7)

[0303] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0304] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0305] Based on an image of the object captured by a second camera facing a direction corresponding to the first camera, identification information obtained by identifying the captured object is output.

[0306] <Note 2> (1)

[0308] An information processing device, wherein the information processing device comprises:

[0309] a first processor, which outputs coordinate values ​​of at least two coordinate axes of a point representing a location of the captured object in a three-dimensional orthogonal coordinate system according to an image of the object captured by the first camera;

[0310] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0311] a third processor that associates the coordinate value output from the first processor with the identification information output from the second processor. (2)

[0313] The information processing device according to (1), wherein:

[0314] The first processor outputs the coordinate values ​​of at least two diagonal points among vertices of a polygon surrounding the outline of the object recognized from the image captured by the first camera. (3)

[0316] The information processing device according to (2), wherein:

[0317] The first processor outputs the coordinate values ​​of a plurality of vertices of a polygon surrounding an outline of the object recognized from the image captured by the first camera. (4)

[0319] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0320] outputting coordinate values ​​of at least two coordinate axes of a point representing the location of the captured object in a three-dimensional orthogonal coordinate system according to an image of the object captured by the first camera,

[0321] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0322] The coordinate value is associated with the identification information. (5)

[0324] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0325] outputting coordinate values ​​of at least two coordinate axes of a point representing the location of the captured object in a three-dimensional orthogonal coordinate system according to an image of the object captured by the first camera,

[0326] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0327] The coordinate value is associated with the identification information.

[0328] <Addendum 3> (1)

[0330] An information processing device, wherein the information processing device comprises:

[0331] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera;

[0332] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0333] A third processor is configured to associate the point information output from the first processor with the identification information output from the second processor, and to control the automatic driving of the moving object based on the point information and the identification information. (2)

[0335] The information processing device according to (1), wherein:

[0336] The third processor

[0337] calculating a control variable for controlling the automatic driving of the moving body based on the detection information detected by the detection unit,

[0338] Based on the calculated control variable as well as the point information and the identification information, the automatic driving of the moving object is controlled. (3)

[0340] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0341] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0342] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0343] The point information is associated with the identification information, and the automatic driving of the moving object is controlled based on the point information and the identification information. (4)

[0345] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0346] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0347] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0348] The point information is associated with the identification information, and the automatic driving of the moving object is controlled based on the point information and the identification information.

[0349] <Note 4> (1)

[0351] An information processing device, wherein the information processing device comprises:

[0352] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera;

[0353] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0354] a third processor that associates the point information output from the first processor with the identification information output from the second processor,

[0355] The frame rate of the first camera is greater than the frame rate of the second camera. (2)

[0357] The information processing device according to (1), wherein:

[0358] The frame rate of the first camera is more than 10 times the frame rate of the second camera. (3)

[0360] The information processing device according to (2), wherein:

[0361] The frame rate of the first camera is above 100 frames per second, and the frame rate of the second camera is 10 frames per second. (4)

[0363] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0364] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0365] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera having a frame rate lower than that of the first camera and facing a direction corresponding to the first camera,

[0366] The point information is associated with the identification information. (5)

[0368] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0369] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0370] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera having a frame rate lower than that of the first camera and facing a direction corresponding to the first camera,

[0371] The point information is associated with the identification information.

[0372] <Addendum 5> (1)

[0374] An information processing device, wherein the information processing device comprises:

[0375] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera;

[0376] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0377] a third processor that associates the point information output from the first processor with the identification information output from the second processor,

[0378] The first processor calculates a degree of danger related to movement of the moving object as a score related to an external environment for a predetermined moving object based on detection information detected by a detection unit and the point information. (2)

[0380] The information processing device according to (1), wherein:

[0381] The frame rate of the first camera is variable.

[0382] The first processor changes the frame rate of the first camera according to the calculated risk level. (3)

[0384] The information processing device according to (1) or (2), wherein:

[0385] The risk level indicates how dangerous a place the moving object will travel in the future. (4)

[0387] An information processing device, wherein the information processing device comprises:

[0388] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera;

[0389] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0390] a third processor that associates the point information output from the first processor with the identification information output from the second processor,

[0391] The third processor calculates a degree of danger related to movement of the moving object as a score related to the external environment for a predetermined moving object based on the detection information detected by the detection unit and the point information. (5)

[0393] The information processing device according to (4), wherein:

[0394] The frame rate of the first camera is variable.

[0395] The third processor outputs an instruction to the first processor to change the frame rate of the first camera according to the calculated risk level. (6)

[0397] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0398] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0399] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0400] Associating the point information with the identification information,

[0401] Based on the detection information detected by the detection unit and the point information, a degree of danger related to the movement of the moving object is calculated as a score related to the external environment for a predetermined moving object. (7)

[0403] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0404] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0405] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0406] Associating the point information with the identification information,

[0407] Based on the detection information detected by the detection unit and the point information, a degree of danger related to the movement of the moving object is calculated as a score related to the external environment for a predetermined moving object.

[0408] <Addendum 6> (1)

[0410] An information processing device, wherein the information processing device comprises:

[0411] a first processor, which outputs point information obtained by capturing the photographed object as a point based on at least one of a visible light image and an infrared image of the object photographed by the first camera;

[0412] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0413] and a third processor that associates the point information output from the first processor with the identification information output from the second processor. (2)

[0415] The information processing device according to (1), wherein:

[0416] When the object cannot be captured from the visible light image of the object captured by the visible light camera included in the first camera due to specified factors, the first processor outputs the point information based on the infrared image of the object captured by the infrared camera included in the first camera. (3)

[0418] The information processing device according to (2), wherein:

[0419] The first processor synchronizes a timing of capturing the visible light image through the visible light camera with a timing of capturing the infrared image through the infrared camera. (4)

[0421] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0422] Based on at least one of a visible light image and an infrared image of an object captured by a first camera, outputting point information obtained by capturing the captured object as a point,

[0423] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0424] The point information is associated with the identification information. (5)

[0426] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0427] Based on at least one of a visible light image and an infrared image of an object captured by a first camera, outputting point information obtained by capturing the captured object as a point;

[0428] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera;

[0429] The point information is associated with the identification information.

[0430] <Addendum 7> (1)

[0432] An information processing device, wherein the information processing device comprises:

[0433] a first processor that outputs point information obtained by capturing the photographed object as a point based on an image of the object photographed by the first camera and a radar signal of a reflected wave based on an electromagnetic wave irradiated onto the object by a radar and reflected from the object;

[0434] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0435] and a third processor that associates the point information output from the first processor with the identification information output from the second processor. (2)

[0437] The information processing device according to (1), wherein:

[0438] The first processor synchronizes a timing at which the first camera captures the image with a timing at which the radar acquires three-dimensional point cloud data of the object based on the radar signal. (3)

[0440] The information processing device according to (1) or (2), wherein:

[0441] The number of images per unit time taken by the first camera and the number of three-dimensional point cloud data per unit time acquired by the radar are greater than the number of images per unit time taken by the second camera. (4)

[0443] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0444] Outputting point information obtained by capturing the photographed object as a point based on an image of the object photographed by the first camera and a radar signal of a reflected wave based on an electromagnetic wave irradiated onto the object by the radar and reflected from the object,

[0445] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0446] The point information is associated with the identification information. (5)

[0448] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0449] Outputting point information obtained by capturing the photographed object as a point based on an image of the object photographed by the first camera and a radar signal of a reflected wave based on an electromagnetic wave irradiated onto the object by the radar and reflected from the object,

[0450] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0451] The point information is associated with the identification information.

[0452] <Addendum 8> (1)

[0454] An information processing device, wherein the information processing device comprises:

[0455] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera;

[0456] a second processor configured to output label information indicating a category of the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0457] a third processor that associates the point information output from the first processor with the tag information output from the second processor. (2)

[0459] The information processing device according to (1), wherein:

[0460] The third processor associates the position information of the object indicated by the point information with the tag information related to the object existing at the position indicated by the position information. (3)

[0462] The information processing device according to (2), wherein:

[0463] The third processor associates the point information output from the first processor at the same timing as the timing at which the second processor outputs the tag information with the tag information. (4)

[0465] The information processing device according to (2) or (3), wherein:

[0466] When new point information is input from the first processor after the point information and the tag information are associated, the third processor also associates the new point information with the tag information. (5)

[0468] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0469] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0470] outputting label information indicating a category of the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0471] The point information is associated with the tag information. (6)

[0473] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0474] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0475] outputting label information indicating a category of the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0476] The point information is associated with the tag information.

[0477] <Addendum 9> (1)

[0479] A cooling execution device, wherein the cooling execution device comprises:

[0480] an acquisition unit that acquires a detection result of the object by an information processing device, the information processing device outputting point information obtained by capturing the object as a point and identification information obtained by identifying the object based on images of the object captured by a plurality of cameras facing corresponding directions, and associating the point information with the identification information; and

[0481] An execution unit that executes cooling of the information processing device based on the detection result acquired by the acquisition unit. (2)

[0483] The cooling execution device according to (1), wherein the cooling execution device comprises a prediction unit,

[0484] The prediction unit predicts the operating status of the information processing device based on the detection result acquired by the acquisition unit,

[0485] The execution unit executes cooling of the information processing device based on a prediction result of the operating condition of the information processing device by the prediction unit. (3)

[0487] The cooling actuator according to (2), wherein:

[0488] The prediction unit predicts a temperature change of the information processing device.

[0489] The execution unit executes cooling of the information processing device using a cooling mechanism according to a prediction result of the temperature change of the information processing device by the prediction unit. (4)

[0491] The cooling actuator according to any one of (1) to (3), wherein:

[0492] The detection result acquired by the acquisition unit is the point information. (5)

[0494] A cooling execution method, wherein the cooling execution method is executed by a computer to perform the following processing:

[0495] obtaining a detection result of the object by an information processing device, wherein the information processing device outputs point information obtained by capturing the object as a point and identification information obtained by identifying the object based on images of the object captured by multiple cameras facing corresponding directions, and associates the point information with the identification information,

[0496] Based on the acquired detection result, the information processing device is cooled down. (6)

[0498] A cooling execution program, wherein the cooling execution program is used to cause a computer to execute the following processing:

[0499] obtaining a detection result of the object by an information processing device, wherein the information processing device outputs point information obtained by capturing the object as a point and identification information obtained by identifying the object based on images of the object captured by multiple cameras facing corresponding directions, and associates the point information with the identification information,

[0500] Based on the acquired detection result, the information processing device is cooled down.

[0501] <Addendum 10> (1)

[0503] An information processing device, wherein the information processing device comprises:

[0504] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera;

[0505] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0506] a third processor that associates the point information output from the first processor with the identification information output from the second processor,

[0507] The first processor derives, from the image of the object captured by the first camera, a coordinate value of a point indicating an existing position of the object in a depth direction of the object in a three-dimensional orthogonal coordinate system as the point information. (2)

[0509] The information processing device according to (1), wherein:

[0510] The first processor derives the coordinate value in the depth direction as the point information from the image of the object captured by the plurality of the first cameras. (3)

[0512] The information processing device according to (1) or (2), wherein:

[0513] The first processor derives the coordinate values ​​of the object in the width direction, the height direction, and the depth direction as the point information based on the image of the object captured by the first camera and the radar signal of the reflected wave reflected from the object based on the electromagnetic wave irradiated on the object by the radar. (4)

[0515] An information processing device according to any one of (1) to (3), wherein:

[0516] The first processor derives the coordinate values ​​of the object in the width direction, the height direction and the depth direction as the point information based on the image of the object captured by the first camera and the result of capturing the structured light irradiated to the object by the irradiation device. (5)

[0518] An information processing device according to any one of (1) to (4), wherein:

[0519] The first processor derives the coordinate value in the depth direction of the second time point as the point information based on the coordinate values ​​in the width direction, height direction and depth direction of the object in the three-dimensional orthogonal coordinate system at the first time point, and the coordinate values ​​in the width direction and the height direction at the second time point, and the second time point is the next time point of the first time point. (6)

[0521] An information processing device, wherein the information processing device comprises:

[0522] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera;

[0523] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the first camera; and

[0524] a third processor that associates the point information output from the first processor with the identification information output from the second processor,

[0525] The third processor derives, from the image of the object captured by the first camera, a coordinate value of a point indicating an existing position of the object in a depth direction of the object in a three-dimensional orthogonal coordinate system as the point information. (7)

[0527] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0528] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0529] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera;

[0530] Associating the point information with the identification information,

[0531] The coordinate value of a point indicating the existence position of the object in the depth direction of the object in a three-dimensional orthogonal coordinate system is derived as the point information from the image of the object captured by the first camera. (8)

[0533] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0534] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the first camera,

[0535] outputting identification information obtained by identifying the photographed object according to an image of the object photographed by a second camera facing a direction corresponding to the first camera,

[0536] Associating the point information with the identification information,

[0537] The coordinate value of a point indicating the existence position of the object in the depth direction of the object in a three-dimensional orthogonal coordinate system is derived as the point information from the image of the object captured by the first camera.

[0538] <Addendum 11> (1)

[0540] An information processing device, wherein the information processing device comprises:

[0541] a first processor, wherein the first processor outputs point information obtained by capturing the captured object as a point according to an image of the object captured by the event camera;

[0542] a second processor configured to output identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the event camera; and

[0543] and a third processor that associates the point information output from the first processor with the identification information output from the second processor. (2)

[0545] The information processing device according to (1), wherein:

[0546] In a case where the object cannot be captured from the visible light image of the object captured by the visible light camera due to a prescribed factor, the first processor outputs the point information based on the image of the object captured by the event camera. (3)

[0548] The information processing device according to (2), wherein:

[0549] The predetermined factor includes at least one of a case where the moving speed of the object is greater than or equal to a predetermined value and a case where a change in the amount of ambient light per unit time is greater than or equal to a predetermined value. (4)

[0551] An information processing device according to any one of (1) to (3), wherein:

[0552] The event camera is a camera that outputs an event image, and the event image represents the difference between an image captured at a current moment and an image captured at a previous moment. (5)

[0554] An information processing method, wherein the information processing method is performed by a computer to perform the following processing:

[0555] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the event camera,

[0556] outputting identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the event camera,

[0557] The point information is associated with the identification information. (6)

[0559] An information processing program, wherein the information processing program is used to cause a computer to execute the following processing:

[0560] Outputting point information obtained by capturing the captured object as a point according to an image of the object captured by the event camera,

[0561] outputting identification information obtained by identifying the photographed object based on an image of the object photographed by a second camera facing a direction corresponding to the event camera,

[0562] The point information is associated with the identification information.

[0563] It should be noted that the disclosures of the following Japanese patent applications are incorporated herein by reference in their entirety. In addition, all documents, patent applications, and technical standards described in this specification are incorporated herein by reference as if each document, patent application, and technical standard were specifically and separately described as being incorporated by reference.

[0564] Japanese Patent Application No. 2022-170165 Japanese Patent Application No. 2022-172777 Japanese Patent Application No. 2022-175679 Japanese Patent Application No. 2022-181362 Japanese Patent Application No. 2022-182131 Japanese Patent Application No. 2022-186040 Japanese Patent Application No. 2022-187648 Japanese Patent Application No. 2022-187649 Japanese Patent Application No. 2022-189546 Japanese Patent Application No. 2022-210884 Japanese Patent Application No. 2023-036967 Japanese Patent Application No. 2023-036975 Japanese Patent Application No. 2023-078024 Japanese Patent Application No. 2023-080388

Claims

1. A cooling execution device, in, The cooling execution device comprises: an acquisition unit, the acquisition unit acquiring a detection result of an object related to the operation of a control device, the control device being used to control the automatic driving of a vehicle and being mounted on the vehicle; as well as An execution unit executes cooling in the control device based on the detection result.

2. The cooling actuator according to claim 1, in, As the detection result, the acquisition unit extracts points used to represent the existence position of the object from a 1-frame image of the object, and acquires motion information at a frame rate of more than 100 frames per second, wherein the motion information represents the movement of the points used to represent the existence position of the object along a specified coordinate axis.

3. The cooling execution device according to claim 2, in, The cooling execution device further includes a prediction unit. The prediction unit predicts the operation of the control device using the detection result. The prediction unit predicts the operation of the control device using a learning model generated by machine learning, and the machine learning uses the detection result and the operating status of the control device when the detection result is obtained as learning data.

4. The cooling execution device according to claim 3, in, The prediction unit further predicts temperature changes of each of a plurality of locations in the control device. The execution unit controls cooling of the portion in the control device.

5. A cooling execution method, in, The cooling execution method is performed by a computer to execute the following processing: Acquire a detection result of an object related to the operation of a control device, the control device is used to control the automatic driving of a vehicle and is mounted on the vehicle, Based on the detection result, cooling in the control device is performed.

6. A cooling execution procedure, in, The cooling execution program is used to make the computer execute the following processing: Acquire a detection result of an object related to the operation of a control device, the control device is used to control the automatic driving of a vehicle and is mounted on the vehicle, Based on the detection result, cooling in the control device is performed.

7. The cooling actuator according to claim 1, in, The cooling execution device comprises: an acquisition unit, the acquisition unit acquiring a detection result of the object by an information processing device, the information processing device outputting point information obtained by capturing the object as a point and identification information obtained by identifying the object based on images of the object captured by a plurality of cameras facing corresponding directions, and associating the point information with the identification information; as well as An execution unit that executes cooling of the information processing device based on the detection result acquired by the acquisition unit.

8. The cooling execution device according to claim 7, in, The execution unit stops cooling the information processing device when a predetermined condition is satisfied.

9. The cooling execution device according to claim 8, in, The execution unit stops cooling the information processing device when the predetermined condition that the object detected by the information processing device is no longer detected is satisfied.

10. The cooling execution device according to claim 8, in, The execution unit stops cooling the information processing device when the predetermined condition that the object detected by the information processing device is approaching outside a detection range is satisfied.

11. The cooling implementation device according to any one of claims 8 to 10, in, The execution unit stops cooling the information processing device after a predetermined time has passed when a predetermined condition is satisfied.

12. The cooling execution device according to claim 8, in, The cooling execution device includes a prediction unit, The prediction unit predicts the operating status of the information processing device based on the detection result acquired by the acquisition unit, The execution unit executes cooling of the information processing device based on a prediction result of the operating condition of the information processing device by the prediction unit.

13. The cooling execution device according to claim 12, in, The prediction unit predicts a temperature change of the information processing device. The execution unit executes cooling of the information processing device using a cooling mechanism according to a prediction result of the temperature change of the information processing device by the prediction unit.

14. The cooling execution device according to claim 12, in, The prediction unit predicts a period when the temperature of the information processing device is below a threshold value. The execution unit stops cooling the information processing device when the predetermined condition that the time period predicted by the prediction unit has arrived is satisfied.

15. The cooling execution device according to claim 8, in, The detection result acquired by the acquisition unit is the point information.

16. The cooling execution device according to claim 7, in, The cooling execution device comprises: a prediction unit configured to predict an operating condition and a temperature change of the information processing device based on the detection result acquired by the acquisition unit; and An execution unit, which executes cooling of the information processing device based on the prediction result of the prediction unit on the working condition of the information processing device, and adjusts the cooling amount of the information processing device based on the prediction result of the prediction unit on the temperature change of the information processing device.

17. The cooling execution device according to claim 16, in, The prediction unit further predicts a peak value of the temperature of the information processing device. The execution unit executes cooling of the information processing device with a cooling amount corresponding to the peak value of the temperature predicted by the prediction unit.

18. The cooling execution device according to claim 17, in, The execution unit executes cooling of the information processing device so that the cooling amount corresponding to the peak value of the temperature is reached before the temperature predicted by the prediction unit reaches the peak value.

19. The cooling execution device according to claim 17, in, The information processing device is mounted on a mobile body. When the object is moving toward the moving body according to the detection result, the prediction unit predicts that the temperature of the information processing device is increasing. When the prediction unit predicts a temperature rise of the information processing device, the execution unit increases the amount of cooling of the information processing device.

20. A cooling execution method, in, The cooling execution method is performed by a computer to execute the following processing: Acquiring a detection result of an object by an information processing device, the information processing device outputting point information obtained by capturing the object as a point and identification information obtained by identifying the object based on images of the object captured by multiple cameras facing corresponding directions, and associating the point information with the identification information, Based on the acquired detection result, the information processing device is cooled down.

21. A cooling execution program, in, The cooling execution program is used to make the computer execute the following processing: Acquiring a detection result of an object by an information processing device, the information processing device outputting point information obtained by capturing the object as a point and identification information obtained by identifying the object based on images of the object captured by multiple cameras facing corresponding directions, and associating the point information with the identification information, Based on the acquired detection result, the information processing device is cooled down.

22. A vehicle, in, The vehicle is provided with a control device for controlling automatic driving, and the vehicle comprises: a power unit, the power unit comprising an electric motor, an acquisition unit, the acquisition unit acquiring a detection result of an object related to the operation of the control device, an execution unit that executes cooling in the control device based on the detection result, The control device controls each part of the vehicle including the cooling using the regenerative power generated by the electric motor when the vehicle is decelerated according to the detection result.

23. The vehicle according to claim 22, in, The control device controls the deceleration of the vehicle so that each part of the vehicle is controlled only by the regenerative electric power from the start of deceleration of the vehicle in accordance with the detection result until the vehicle stops.

24. The vehicle according to claim 22, in, The vehicle is provided with a brake actuator, The brake actuator performs a braking operation of the vehicle, The control device controls the brake actuator using the regenerative electric power when decelerating the vehicle in accordance with the detection result.

25. A vehicle according to claim 23 or 24, in, The vehicle includes a steering actuator, The steering actuator performs a steering operation of the vehicle, The control device controls the steering actuator using the regenerative electric power when decelerating the vehicle in accordance with the detection result.

26. The vehicle according to claim 22, in, As the detection result, the acquisition unit extracts points used to represent the existence position of the object from a 1-frame image of the object, and acquires motion information at a frame rate of more than 100 frames per second, wherein the motion information represents the movement of the points used to represent the existence position of the object along a specified coordinate axis.

27. The vehicle according to claim 26, in, The vehicle further includes a prediction unit. The prediction unit predicts the operation of the control device using the detection result. The prediction unit predicts the operation of the control device using a learning model generated by machine learning, and the machine learning uses the detection result and the operating status of the control device when the detection result is obtained as learning data.

28. The vehicle according to claim 27, in, The prediction unit further predicts temperature changes of each of a plurality of locations in the control device. The execution unit controls cooling of the portion in the control device.

Images