Automatic driving control device and automatic driving control method
By designing the autonomous driving control device, using information acquisition, monitoring and control components, the amount of autonomous driving control that reduces the sensor's impact is derived based on the machine learning model, the automatic driving control problem caused by the reduction of sensor reliability is solved, and the vehicle is realized Stable driving in various situations.
Patent Information
- Application Number
- CN201980099843.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-02
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, when the reliability of any sensor in multiple sensors is reduced, the amount of automatic driving control is not suitable for vehicle autonomous driving control, and the calculation formula derived by the machine learning model is not theoretically determined, making it difficult to solve the environmental speculation problem in the case of failure. .
An autonomous driving control device is designed, and the information acquisition unit obtains vehicle peripheral information from multiple sensors through the information acquisition unit. The control quantity derivation unit derives the automatic driving control quantity based on the machine learning model, and removes the influence when the reliability decreases. The control unit Output the control amount after the influence is removed.
Even when the reliability of sensor acquisition information is reduced, the amount of autonomous driving control suitable for the vehicle can be output to ensure the safe driving of the vehicle.
Smart Images

Figure CN114286772B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an automatic driving control device and an automatic driving control method for performing automatic driving control of a vehicle. Background Art
[0002] Information about the surroundings of a vehicle (hereinafter referred to as "vehicle surrounding information") can be obtained from a plurality of sensors. In the past, there was a technology that derives and outputs various control quantities (hereinafter referred to as "autonomous driving control quantities") required for the automatic driving control of a vehicle by inputting a plurality of vehicle surrounding information respectively outputted from such a plurality of sensors into a model that has completed machine learning (hereinafter referred to as "machine learning model").
[0003] Here, Patent Document 1 discloses a control device that uses a map of the surrounding environment generated based on multiple images captured by a compound eye camera to identify the surrounding environment during the execution of the automatic driving control of the vehicle. For example, when any of the two vehicle-mounted cameras constituting the compound eye camera fails to function properly, the control device estimates the surrounding environment based on the image captured by a normally operating vehicle-mounted camera.
[0004] Prior art literature
[0005] Patent Literature
[0006] Patent Document 1:
[0007] Japanese Patent Publication No. 2019-34664 Summary of the invention
[0008] Technical problem to be solved by the invention
[0009] The above-mentioned technology of using a machine learning model to derive the amount of autonomous driving control has the following problem: when the reliability of information obtained from any sensor among multiple sensors decreases, the amount of autonomous driving control may no longer be suitable for the autonomous driving control of the vehicle.
[0010] The control device disclosed in Patent Document 1 estimates the surrounding environment based on the image captured by the normally functioning on-board camera when any of the multiple on-board cameras malfunctions, but the estimation is specifically performed using a theoretically determined calculation formula. In this regard, since the machine learning model is not derived using a theoretically determined calculation formula, the technology in the control device disclosed in Patent Document 1 cannot be used as a means to solve the above problem.
[0011] The present invention is completed to solve the above-mentioned problem, and its purpose is to provide an automatic driving control device, which derives and outputs automatic driving control quantities based on multiple vehicle surrounding information output from multiple sensors and a machine learning model. Even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control quantity suitable for the automatic driving control of the vehicle can be output.
[0012] Technical means for solving technical problems
[0013] The automatic driving control device involved in the present invention includes: an information acquisition unit, which acquires multiple vehicle surrounding information output from multiple sensors respectively; a control amount derivation unit, which derives the automatic driving control amount based on the multiple vehicle surrounding information acquired by the information acquisition unit and a machine learning model, and outputs the automatic driving control amount; a monitoring unit, which determines whether the reliability of any vehicle surrounding information among the multiple vehicle surrounding information acquired by the information acquisition unit is reduced; and a control unit, which, when the monitoring unit determines that the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, controls the control amount derivation unit, so that the control amount derivation unit outputs the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability.
[0014] Effects of the Invention
[0015] According to the present invention, even when the reliability of any of a plurality of vehicle surrounding information outputted from a plurality of sensors is reduced, an automatic driving control amount suitable for automatic driving control of the vehicle can be outputted. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 This is a diagram showing a configuration example of a vehicle equipped with the automatic driving control device according to the first embodiment.
[0017] Figure 2 This is a diagram showing a structural example of an automatic driving control device involved in Implementation Example 1.
[0018] Figure 3 This is a flowchart for explaining the operation of the automatic driving control device involved in Implementation Example 1.
[0019] Figure 4 This is a diagram showing a structural example of an automatic driving control device involved in Implementation Example 2.
[0020] Figure 5 This is a flowchart for illustrating the operation of the automatic driving control device involved in Implementation Example 2.
[0021] Figure 6 This is a diagram showing a structural example of an automatic driving control device involved in Implementation Example 3.
[0022] FIG. 7 is a diagram for explaining an example of a captured image in which the brightness of pixels is different in Embodiment 3. Fig. 7A is an example of a captured image when all pixels of the captured image have brightness sufficient to recognize an object in the captured image, Figure 7B is an example of a captured image when all pixels of the captured image have a brightness that makes the image so dark that an object in the captured image cannot be recognized, Figure 7C This is an example of a captured image in which all pixels of the captured image have a brightness that makes the image so bright that an object in the captured image cannot be recognized.
[0023] Figure 8 This is a flowchart for illustrating the operation of the automatic driving control device involved in the third embodiment.
[0024] Fig. 9 This is a diagram showing a structural example of an automatic driving control device involved in Implementation Example 4.
[0025] Fig.10 This is a flowchart for illustrating the operation of the automatic driving control device involved in embodiment 4.
[0026] Fig.11 This is a diagram showing a structural example of an automatic driving control device involved in Implementation Example 5.
[0027] Fig.12 This is a diagram showing an example of an image of a display on which a notification control unit displays notification information in the fifth embodiment.
[0028] Fig.13 This is a diagram showing another image example of a display on which the notification control unit displays notification information in the fifth embodiment.
[0029] Fig.14 This is a diagram showing a structural example of an automatic driving control device involved in Implementation Example 6.
[0030] Fig.15 This is a flowchart for illustrating the operation of the automatic driving control device involved in embodiment 6.
[0031] Fig.16A , Fig. 16B This is a diagram showing an example of the hardware configuration of the automatic driving control device according to the first to sixth embodiments.
[0032] Fig.17 is a diagram showing an example of the structure of an automatic driving control system, which is assumed to use Figure 2The automatic driving control device according to the first embodiment described above is included in a server. DETAILED DESCRIPTION
[0033] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
[0034] Implementation method 1.
[0035] Figure 1 This is a diagram showing a configuration example of a vehicle 100 equipped with the automatic driving control device 1 according to the first embodiment.
[0036] The automatic driving control device 1 according to the first embodiment is a device installed in a vehicle 100 that can automatically travel even without a human driving operation. Figure 1 As shown, in addition to the automatic driving control device 1 , a sensor, a vehicle control unit 3 , and a controlled device 4 are mounted in the vehicle 100 .
[0037] A plurality of sensors that output information about the periphery of the vehicle 100 are mounted on the vehicle 100. The information outputted by the plurality of sensors is information about other vehicles existing around the vehicle 100, information about obstacles other than other vehicles existing around the vehicle 100, information about the state of traffic signals, information about lanes, information about terrain, or information about road signs, etc. The information about lanes includes, for example, white lines or road signs.
[0038] In the first embodiment, the sensor includes the camera 21 and the millimeter wave radar 22 .
[0039] The camera 21 captures the surroundings of the vehicle 100 , for example, the front of the vehicle 100 . The camera 21 outputs a captured image of the surroundings of the vehicle 100 to the automatic driving control device 1 .
[0040] The millimeter wave radar 22 measures the distance from the vehicle 100 to an object existing around the vehicle 100. The millimeter wave radar 22 outputs information related to the measured distance from the vehicle 100 to the object (hereinafter referred to as "distance information") to the automatic driving control device 1. The automatic driving control device 1 derives the automatic driving control amount required for the automatic driving control of the vehicle 100 based on at least the captured image output from the camera 21 and the distance information output from the millimeter wave radar 22.
[0041] In the first embodiment, information output from sensors and used for deriving the automatic driving control amount in the automatic driving control device 1, such as the captured image or distance information, is collectively referred to as "vehicle surrounding information". The vehicle surrounding information is information used for deriving the automatic driving control amount in the automatic driving control device 1, and can include various information related to the surroundings of the vehicle 100.
[0042] In Implementation 1, as a premise, for one or more sensors among the multiple sensors, for example, the possibility of failure is so low that it can be ignored, and the vehicle surrounding information output from the sensor does not substantially cause problems. On the other hand, as a premise, for another one or more sensors among the multiple sensors, for example, compared with the above-mentioned part of the sensors, the possibility of failure is higher, and the vehicle surrounding information output from the other sensors is also more likely to have problems. Specifically, as a premise, for the camera 21 and the millimeter-wave radar 22 included in the sensor involved in Implementation 1, for example, the possibility of failure is so low that it can be ignored, and the image captured by the camera 21 basically does not cause problems. On the other hand, as a premise, for the millimeter-wave radar 22, compared with the camera 21, for example, the possibility of failure is high, and the distance information output by the millimeter-wave radar 22 is also more likely to have problems.
[0043] In Embodiment 1, as described later, the vehicle surrounding information is used as input data for a machine learning model for deriving an autonomous driving control amount. In Embodiment 1, as input data for deriving an autonomous driving control amount suitable for autonomous driving control of the vehicle 100, the degree to which the vehicle surrounding information is reliable is referred to as the "reliability" of the vehicle surrounding information.
[0044] The automatic driving control device 1 derives the automatic driving control amount based on the vehicle surrounding information output from the sensor. The details of the derivation of the automatic driving control amount by the automatic driving control device 1 will be described later together with the configuration example of the automatic driving control device 1.
[0045] The automatic driving control device 1 outputs the derived automatic driving control amount to the vehicle control unit 3 mounted on the vehicle 100 .
[0046] The vehicle control unit 3 controls the vehicle 100 based on the automatic driving control amount output from the automatic driving control device 1. Specifically, the vehicle control unit 3 controls the control target device 4 to automatically drive the vehicle 100. The control target device 4 is a device mounted on the vehicle 100 and operates based on the control of the vehicle control unit 3 to automatically drive the vehicle 100. The control target device 4 is, for example, an accelerator, a brake, a steering, a gear, or a light.
[0047] The automatic driving control amount output by the automatic driving control device 1 may be a specific control amount of each control object device 4 such as a brake, an accelerator or a steering, or may be information of a driving trajectory composed of multiple latitude and longitude values in a time series. When the automatic driving control amount is driving trajectory information, the vehicle control unit 3 calculates the specific control amount of each control object device 4 so that the vehicle 100 automatically drives according to the driving trajectory, and controls each control object device 4 based on the calculated control amount.
[0048] The details of the automatic driving control device 1 according to the first embodiment will be described.
[0049] Figure 2 This is a diagram showing a configuration example of the automatic driving control device 1 according to the first embodiment.
[0050] The automatic driving control device 1 includes an information acquisition unit 11, a control amount derivation unit 12, a machine learning model 13, a monitoring unit 14 and a control unit 15. The control amount derivation unit 12 includes a first control amount derivation unit 121, a second control amount derivation unit 122 and a selection unit 123. The machine learning model 13 includes a first machine learning model 131 and a second machine learning model 132.
[0051] The information acquisition unit 11 acquires a plurality of vehicle surrounding information outputted from a plurality of sensors; specifically, the information acquisition unit 11 acquires the captured image captured by the camera 21 and the distance information measured by the millimeter wave radar 22 as the vehicle surrounding information. The information acquisition unit 11 outputs the acquired vehicle surrounding information to the control amount derivation unit 12 and the monitoring unit 14.
[0052] The control amount derivation unit 12 derives the automatic driving control amount of the vehicle 100 based on the plurality of vehicle surrounding information acquired by the information acquisition unit 11 and the machine learning model 13, and outputs the automatic driving control amount. In the first embodiment, specifically, the control amount derivation unit 12 associates the derived automatic driving control amount with the information that specifies the control object device 4 that is the control object, and outputs it to the vehicle control unit 3.
[0053] The first control amount derivation unit 121 of the control amount derivation unit 12 derives the first automatic driving control amount based on all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 and the first machine learning model 131. Specifically, the first control amount derivation unit 121 derives the first automatic driving control amount based on the captured image acquired by the information acquisition unit 11 from the camera 21, the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22, and the first machine learning model 131. The first machine learning model 131 will be described later.
[0054] The first control amount derivation unit 121 outputs the derived first automatic driving control amount to the selection unit 123 .
[0055] The second control amount derivation unit 122 of the control amount derivation unit 12 derives the second automatic driving control amount based on a part of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 and the second machine learning model 132. Specifically, the second control amount derivation unit 122 derives the second automatic driving control amount based on the vehicle surrounding information other than the vehicle surrounding information whose reliability is reduced among the plurality of vehicle surrounding information and the second machine learning model 132.
[0056] As described above, in Embodiment 1, as a premise, the captured image of the camera 21 does not cause any problem in essence, but the distance information of the millimeter wave radar 22 is more likely to cause a problem. Therefore, as described later, the second machine learning model 132 in Embodiment 1 specifically uses only the captured image acquired by the information acquisition unit 11 as input to derive the second automatic driving control amount.
[0057] The second control amount derivation unit 122 outputs the derived second automatic driving control amount to the selection unit 123 .
[0058] The selection unit 123 selects which of the first automatic driving control amount and the second automatic driving control amount to output. In the first embodiment, the selection unit 123 specifically outputs the selected automatic driving control amount to the vehicle control unit 3.
[0059] As described later, when the monitoring unit 14 determines that the reliability of all the plurality of vehicle periphery information has not decreased, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the first automatic driving control amount. When the control amount derivation unit 12 is controlled by the control unit 15 to output the first automatic driving control amount, the selection unit 123 selects and outputs the first automatic driving control amount.
[0060] In addition, when the monitoring unit 14 determines that the reliability of the vehicle surrounding information other than a part of the plurality of vehicle surrounding information input to the second machine learning model 132 among the plurality of vehicle surrounding information is reduced, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount. When the control amount derivation unit 12 is controlled by the control unit 15 to output the second automatic driving control amount, the selection unit 123 selects and outputs the second automatic driving control amount.
[0061] The machine learning model 13 is a model that has been learned in machine learning. Specifically, the machine learning model 13 is a model that has been pre-learned so as to output the autonomous driving control amount required for the autonomous driving control of the vehicle 100 when the plurality of vehicle surrounding information acquired by the input information acquisition unit 11 is input. The machine learning model 13 is composed of, for example, a neural network.
[0062] In implementation mode 1, the machine learning model 13 includes a first machine learning model 131 and a second machine learning model 132 .
[0063] The first machine learning model 131 takes all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 as input, and outputs a first autonomous driving control amount. In the first embodiment, the first machine learning model 131 takes both the captured image acquired by the information acquisition unit 11 from the camera 21 and the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 as input, and outputs a first autonomous driving control amount.
[0064] The second machine learning model 132 takes as input a part of the plurality of vehicle surrounding information acquired by the information acquisition unit 11, and outputs a second autonomous driving control amount. Specifically, when the vehicle surrounding information other than the vehicle surrounding information with reduced reliability among the plurality of vehicle surrounding information acquired by the information acquisition unit 11 is taken as input, the second machine learning model 132 outputs the second autonomous driving control amount.
[0065] As described above, in Embodiment 1, as a premise, the captured image of the camera 21 does not substantially cause problems, but the distance information of the millimeter wave radar 22 is more likely to cause problems. Therefore, specifically, the second machine learning model 132 in Embodiment 1 outputs the second automatic driving control amount using only the captured image acquired by the information acquisition unit 11 as input.
[0066] In implementation mode 1, if Figure 2 As shown, the machine learning model 13 is provided in the automatic driving control device 1, but this is merely an example, and the machine learning model 13 may also be provided in a location outside the automatic driving control device 1 that the automatic driving control device 1 can refer to.
[0067] The monitoring unit 14 determines whether the reliability of any vehicle surrounding information among the plurality of vehicle surrounding information acquired by the information acquisition unit 11 is reduced. In the first embodiment, as a premise, the image captured by the camera 21 does not substantially cause problems, while the distance information of the millimeter wave radar 22 is more likely to cause problems. Therefore, the monitoring unit 14 of the first embodiment determines whether the reliability of the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 is reduced.
[0068] Specifically, the monitoring unit 14 obtains the distance in the actual space from the vehicle 100 to a certain object existing in the captured image based on the captured image acquired from the camera 21 (hereinafter referred to as the "reference distance"). In addition, the monitoring unit 14 calculates the difference between the reference distance and the distance from the vehicle 100 to the above-mentioned object obtained based on the distance information acquired from the millimeter wave radar 22. Then, the monitoring unit 14 determines whether the calculated difference is below a predetermined threshold value (hereinafter referred to as the "radar determination threshold value"). When it is determined that the calculated difference is below the radar determination threshold value, the monitoring unit 14 determines that the reliability of the distance information has not decreased. On the other hand, when it is determined that the calculated difference is greater than the radar determination threshold value, the monitoring unit 14 determines that the reliability of the distance information has decreased. In addition, as a method for the monitoring unit 14 to obtain the reference distance based on the captured image acquired from the camera 21, any known method can be adopted. As a specific method, for example, there is a method of using a completed learning model obtained by learning using a pair of a captured image of an object and a measured value of the distance from the vehicle 100 to the object in real space as teaching data.
[0069] The monitoring unit 14 outputs information related to the result obtained by determining whether the reliability of any vehicle surrounding information among the multiple vehicle surrounding information acquired by the information acquisition unit 11 is reduced (hereinafter referred to as "monitoring result information") to the control unit 15. For example, when the monitoring unit 14 determines that the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the monitoring unit 14 outputs the monitoring result information indicating that the reliability of the vehicle surrounding information is reduced to the control unit 15. The monitoring result information includes information for determining which vehicle surrounding information has reduced credibility. In embodiment 1, when the monitoring unit 14 determines that the reliability of the distance information acquired from the millimeter wave radar 22 is reduced, the monitoring unit 14 outputs the monitoring result information indicating that the reliability of the distance information is reduced to the control unit 15.
[0070] For example, when the monitoring unit 14 determines that the reliability of any of the plurality of vehicle surrounding information has not decreased, the monitoring unit 14 outputs monitoring result information indicating that the reliability of the vehicle surrounding information has not decreased to the control unit 15 .
[0071] When the monitoring unit 14 determines that the reliability of any vehicle surrounding information among multiple vehicle surrounding information is reduced, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the automatic driving control amount after removing the influence of the vehicle surrounding information determined to have reduced reliability.
[0072] Specifically, when the monitoring unit 14 determines that the reliability of the vehicle surrounding information other than a part of the plurality of vehicle surrounding information input to the second machine learning model 132 among the plurality of vehicle surrounding information is reduced, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount. When the control amount derivation unit 12 is controlled by the control unit 15 to output the second automatic driving control amount, the selection unit 123 of the control amount derivation unit 12 selects and outputs the second automatic driving control amount.
[0073] On the other hand, when the monitoring unit 14 determines that the reliability of all the vehicle surrounding information among the plurality of vehicle surrounding information has not decreased, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the first automatic driving control amount. When the control amount derivation unit 12 is controlled by the control unit 15 to output the first automatic driving control amount, the selection unit 123 of the control amount derivation unit 12 selects and outputs the first automatic driving control amount.
[0074] The operation of the automatic driving control device 1 according to the first embodiment will be described.
[0075] Figure 3 This is a flowchart for explaining the operation of the automatic driving control device 1 according to the first embodiment.
[0076] The information acquisition unit 11 acquires a plurality of vehicle surrounding information outputted from a plurality of sensors. Specifically, the information acquisition unit 11 acquires the captured image captured by the camera 21 and the distance information measured by the millimeter wave radar 22 (step ST301). The information acquisition unit 11 outputs the acquired vehicle surrounding information to the control amount derivation unit 12 and the monitoring unit 14.
[0077] The first control amount derivation unit 121 of the control amount derivation unit 12 derives the first automatic driving control amount based on all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST301 and the first machine learning model 131. Specifically, the first control amount derivation unit 121 derives the first automatic driving control amount based on the captured image acquired by the information acquisition unit 11 from the camera 21 in step ST301, the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 in step ST301, and the first machine learning model 131 (step ST302).
[0078] The first control amount derivation unit 121 outputs the derived first automatic driving control amount to the selection unit 123 .
[0079] The second control amount derivation unit 122 of the control amount derivation unit 12 derives the second automatic driving control amount based on a part of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST301 and the second machine learning model 132. Specifically, the second control amount derivation unit 122 uses only the captured image acquired by the information acquisition unit 11 from the camera 21 in step ST301 as input to derive the second automatic driving control amount (step ST303).
[0080] The second control amount derivation unit 122 outputs the derived second automatic driving control amount to the selection unit 123 .
[0081] The monitoring unit 14 determines whether the reliability of any vehicle surrounding information among the plurality of vehicle surrounding information acquired by the information acquisition unit 11 is reduced. Specifically, first, the monitoring unit 14 acquires a reference distance for an object existing in the captured image based on the captured image acquired by the information acquisition unit 11 from the camera 21 in step ST301 (step ST304).
[0082] Then, the monitoring unit 14 calculates the difference between the reference distance acquired in step ST304 and the distance from the vehicle 100 to the object obtained based on the distance information acquired by the information acquisition unit 11 from the millimeter-wave radar 22 in step ST301, and determines whether the calculated difference is below the radar judgment threshold (step ST305).
[0083] When it is determined in step ST305 that the calculated difference is greater than the radar judgment threshold ("No" in step ST305), the monitoring unit 14 determines that the reliability of the distance information acquired from the millimeter-wave radar 22 is reduced, and outputs the monitoring result information indicating that the reliability of the distance information is reduced to the control unit 15.
[0084] The control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount derived by the second control amount derivation unit 122 in step ST303. The selection unit 123 selects the second automatic driving control amount and outputs it to the vehicle control unit 3 (step ST306).
[0085] When it is determined in step ST305 that the calculated difference is below the radar judgment threshold (when the answer is "yes" in step ST305), the monitoring unit 14 determines that the reliability of the distance information acquired from the millimeter-wave radar 22 has not decreased, and outputs the monitoring result information indicating that the reliability of the distance information has not decreased to the control unit 15.
[0086] The control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the first automatic driving control amount derivation unit 121 in step ST302. The selection unit 123 selects the first automatic driving control amount and outputs it to the vehicle control unit 3 (step ST307).
[0087] When the action of step ST306 or step ST307 is performed, the action of the automatic driving control device 1 returns to step ST301 and repeats the subsequent actions.
[0088] For the above-described actions, when deriving the first automatic driving control amount in step ST302, the first control amount deriving unit 121 does not need to use the captured image and distance information acquired by the information acquiring unit 11 in the previous step ST301. For example, the information acquiring unit 11 may store the acquired vehicle surrounding information in advance in a storage unit (not shown), and the first control amount deriving unit 121 may derive the first automatic driving control amount using the captured image and distance information acquired by the information acquiring unit 11 before the previous step ST301 and stored in the storage unit.
[0089] In addition, when deriving the second automatic driving control amount in step ST303, the second control amount deriving unit 122 does not need to use the captured image acquired by the information acquiring unit 11 in the previous step ST301. For example, the second control amount deriving unit 122 may derive the second automatic driving control amount using the captured image acquired by the information acquiring unit 11 before the previous step ST301 and stored in the storage unit.
[0090] As described above, the automatic driving control device 1 involved in embodiment 1 determines whether the reliability of the distance information output from the millimeter-wave radar 22 is reduced. When it is determined that the reliability of the distance information is reduced, it is possible to continue to derive the automatic driving control amount without using the distance information output from the millimeter-wave radar 22.
[0091] In addition, even if it is determined that the reliability of the distance information obtained from the millimeter wave radar 22 is reduced, the automatic driving control device 1 can continue the automatic driving of the vehicle 100, but the level of automatic driving may be reduced. That is, since the amount of vehicle surrounding information used by the second control amount derivation unit 122 for derivation of the automatic driving control amount is smaller than that of the first control amount derivation unit 121, a difference in the derivation level may occur.
[0092] Specifically, for example, the following difference in derivation level may occur: the first control amount derivation unit 121 is capable of deriving the automatic driving control amount for performing complex controls such as lane changes in congested situations, whereas the second control amount derivation unit 122 can only derive the automatic driving control amount for maintaining driving in the lane in which the vehicle is traveling.
[0093] However, without considering that the reliability of the distance information output from the millimeter wave radar 22 may decrease, more specifically, when the automatic driving control device 1 is configured to include only the first control amount derivation unit 121, when the reliability of the distance information decreases, there is a high possibility that the automatic driving cannot be continued normally. In response to this, the automatic driving control device 1 involved in Embodiment 1 includes a second control amount derivation unit 122, which can derive a second automatic driving control amount based on the captured image acquired from the camera 21 and the second machine learning model 132. When the reliability of the distance information output from the millimeter wave radar 22 decreases, the automatic driving of the vehicle 100 is controlled using the second automatic driving control amount derived by the second control amount derivation unit 122. As a result, even when the reliability of any of the vehicle surrounding information acquired from the plurality of sensors decreases, the automatic driving of the vehicle 100 can be continued although the level is relatively low.
[0094] In the above-mentioned first embodiment, in the automatic driving control device 1, the monitoring unit 14 determines whether the reliability of the distance information has decreased (see Figure 3 Before step ST305), the first control amount derivation unit 121 and the second control amount derivation unit 122 respectively derive the first automatic driving control amount and the second automatic driving control amount (refer to Figure 3Step ST302 and step ST303 of FIG. 1 . However, this is only an example, and the first control amount derivation unit 121 or the second control amount derivation unit 122 may derive the automatic driving control amount by receiving the determination of whether the reliability of the distance information is reduced by the monitoring unit 14. Specifically, for example, when the monitoring unit 14 determines that the reliability of the distance information is reduced, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount. When the control amount derivation unit 12 is controlled by the control unit 15 to output the second automatic driving control amount, the second control amount derivation unit 122 derives the second automatic driving control amount. On the other hand, when the monitoring unit 14 determines that the reliability of the distance information is not reduced, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the first automatic driving control amount. When the control amount derivation unit 12 is controlled by the control unit 15 to output the first automatic driving control amount, the first control amount derivation unit 121 derives the first automatic driving control amount. The control amount derivation unit 12 outputs the first automatic driving control amount derivation by the first control amount derivation unit 121 or the second automatic driving control amount derivation by the second control amount derivation unit 122 to the vehicle control unit 3 based on the control of the control unit 15. In the case of such a configuration, the automatic driving control device 1 does not need to include the selection unit 123.
[0095] In the above embodiment 1, the plurality of vehicle surrounding information is composed of two, the captured image and the distance information, and since it is assumed that only the distance information is more likely to cause problems, the machine learning model used when the reliability of any of the plurality of vehicle surrounding information is reduced is only provided with the second machine learning model 132 that takes the captured image as input. However, among the plurality of vehicle surrounding information, there are sometimes a plurality of vehicle surrounding information for which the possibility of reduced reliability should be considered. In this case, it is assumed that the second machine learning model is provided that uses the vehicle surrounding information obtained by removing any one of the plurality of vehicle surrounding information for which the possibility of reduced reliability should be considered as input for derivation, and the second machine learning model is provided that uses the vehicle surrounding information obtained by removing any combination of the plurality of vehicle surrounding information for which the possibility of reduced reliability should be considered as input for derivation.
[0096] As described above, according to embodiment 1, the automatic driving control device 1 is configured to include: an information acquisition unit 11, which acquires multiple vehicle surrounding information output from multiple sensors (camera 21, millimeter wave radar 22); a control amount derivation unit 12, which derives the automatic driving control amount based on the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the machine learning model 13, and outputs the automatic driving control amount; a monitoring unit 14, which determines whether the reliability of any vehicle surrounding information among the multiple vehicle surrounding information acquired by the information acquisition unit 11 is reduced; and a control unit 15, which controls the control amount derivation unit 12 when the monitoring unit 14 determines that the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, so that the control amount derivation unit 12 outputs the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability.
[0097] Therefore, in the automatic driving control device 1 that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors and the machine learning model 13, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0098] More specifically, in the automatic driving control device 1 involved in implementation mode 1, the control amount derivation unit 12 includes: a first control amount derivation unit 121, which derives a first automatic driving control amount based on all of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the first machine learning model 131; and a second control amount derivation unit 122, which derives a second automatic driving control amount based on a part of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the second machine learning model 132. When the monitoring unit 14 determines that the reliability of the vehicle surrounding information among the multiple vehicle surrounding information other than a part of the multiple vehicle surrounding information input into the second machine learning model 132 is reduced, the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount. Therefore, in the automatic driving control device 1 that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors and the machine learning model 13, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0099] Implementation method 2.
[0100] In the first embodiment, as a premise, the vehicle surrounding information outputted from one or more sensors among the multiple sensors does not substantially cause problems, while on the other hand, the vehicle surrounding information outputted from another one or more sensors among the multiple sensors is more likely to cause problems. Specifically, in the first embodiment, as a premise, of the camera 21 and the millimeter wave radar 22 included in the sensor, the captured image outputted from the camera 21 does not substantially cause problems, while on the other hand, the distance information outputted from the millimeter wave radar 22 is more likely to cause problems.
[0101] In Embodiment 2, it is assumed that at least one of the plurality of sensors is the camera 21. Then, an embodiment will be described in which the automatic driving control device 1a derives an automatic driving control amount suitable for automatic driving control of the vehicle 100, under the premise that the possibility of failure of all of the plurality of sensors is negligibly low and, on the other hand, a problem is more likely to occur in the captured image output from the camera 21 due to, for example, the influence of weather, even when the reliability of the captured image is reduced.
[0102] The automatic driving control device 1 a according to the second embodiment is assumed to be mounted on the vehicle 100 similarly to the automatic driving control device 1 according to the first embodiment.
[0103] In the second embodiment, as described later Figure 4 In that case, the sensors are the camera 21 and the millimeter wave radar 22. As described above, in the second embodiment, as a premise, the possibility of failure of the camera 21 and the millimeter wave radar 22 is low enough to be ignored, while on the other hand, problems are more likely to occur in the captured images output from the camera 21 due to, for example, the influence of weather.
[0104] The automatic driving control device 1a obtains the captured image output from the camera 21 and the distance information output from the millimeter wave radar 22 as vehicle surrounding information, similarly to the first embodiment. In addition, in the second embodiment, the vehicle 100 is equipped with a GNSS (Global Navigation Satellite System) 23, and the automatic driving control device 1a obtains information related to the current position of the vehicle 100 output from the GNSS 23 as information for determining whether the reliability of the captured image is reduced.
[0105] Figure 4 This is a diagram showing a configuration example of an automatic driving control device 1 a according to the second embodiment.
[0106] Regarding the automatic driving control device 1a according to the second embodiment, Figure 2The same structures of the automatic driving control device 1 described are marked with the same reference numerals, and repeated descriptions are omitted. The structure of the automatic driving control device 1a involved in the second embodiment is different from the structure of the automatic driving control device 1 involved in the first embodiment in that it includes a weather determination unit 16. In addition, the specific operations of the second control amount derivation unit 122a and the monitoring unit 14a of the control amount derivation unit 12a are different from the specific operations of the second control amount derivation unit 122 and the monitoring unit 14 of the automatic driving control device 1 involved in the first embodiment.
[0107] As described above, in the second embodiment, as a premise, the captured image output from the camera 21 is more likely to have problems. Therefore, in order to cope with the situation where the reliability of the captured image is reduced, the second machine learning model 132a involved in the second embodiment takes the distance information obtained by the information acquisition unit 11 from the millimeter wave radar 22 as input to output the second automatic driving control amount. The second control amount derivation unit 122a derives the second automatic driving control amount based on the distance information obtained by the information acquisition unit 11 from the millimeter wave radar 22 and the second machine learning model 132a.
[0108] The information acquisition unit 11 acquires, as vehicle periphery information, the image captured by the camera 21 and the distance information measured by the millimeter wave radar 22. The information acquisition unit 11 outputs the acquired vehicle periphery information to the control amount derivation unit 12a and the monitoring unit 14a.
[0109] The weather determination unit 16 acquires information related to the current position of the vehicle 100 from the GNSS 23. In addition, the weather determination unit 16 acquires weather information from a cloud weather server 5 described later via a network such as the Internet. The weather determination unit 16 determines the weather around the vehicle 100 based on the information related to the current position of the vehicle 100 acquired from the GNSS 23 and the weather information acquired from the cloud weather server 5. The weather determination unit 16 determines, for example, whether there is fog or precipitation around the vehicle 100.
[0110] Here, the foggy state or the precipitation state determined by the weather determination unit 16 is a heavy foggy state or a heavy precipitation state in which the reliability of the captured image output from the camera 21 is reduced to the extent that it is not suitable for derivation of the automatic driving control amount. Even if the camera 21 itself is not faulty, when there is heavy fog or heavy precipitation around the vehicle 100, other vehicles existing around the vehicle 100 may not be clearly captured in the captured image output from the camera 21, and therefore such a captured image is not suitable for derivation of the automatic driving control amount.
[0111] The surrounding area of the vehicle 100 that is the target of the weather determination unit 16 for determining whether there is fog or precipitation is predetermined to be, for example, within 1 km around the current position of the vehicle 100 .
[0112] The cloud weather server 5 is a server for distributing information related to weather conditions.
[0113] The weather determination unit 16 outputs the determined information on the weather around the vehicle 100 to the monitoring unit 14 a .
[0114] The monitoring unit 14 a determines whether the reliability of the captured image output from the camera 21 has decreased based on the information on the weather output from the weather determination unit 16 .
[0115] Specifically, for example, when the weather determination unit 16 determines that there is fog or precipitation around the vehicle 100 , the monitoring unit 14 a determines that the reliability of the captured image obtained from the camera 21 has decreased.
[0116] The operation of the automatic driving control device 1 a according to the second embodiment will be described.
[0117] Figure 5 This is a flowchart for explaining the operation of the automatic driving control device 1 a according to the second embodiment.
[0118] The information acquisition unit 11 acquires a plurality of vehicle surrounding information outputted from a plurality of sensors. Specifically, the information acquisition unit 11 acquires the captured image captured by the camera 21 and the distance information measured by the millimeter wave radar 22 as the vehicle surrounding information. In addition, the weather determination unit 16 acquires information related to the current position of the vehicle 100 from the GNSS 23 (step ST501). The information acquisition unit 11 outputs the acquired vehicle surrounding information to the control amount derivation unit 12a and the monitoring unit 14a.
[0119] The first control amount derivation unit 121 of the control amount derivation unit 12a derives the first automatic driving control amount based on all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST501 and the first machine learning model 131. Specifically, the first control amount derivation unit 121 derives the first automatic driving control amount based on the captured image acquired by the information acquisition unit 11 from the camera 21 in step ST501, the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 in step ST501, and the first machine learning model 131 (step ST502). The first control amount derivation unit 121 outputs the derived first automatic driving control amount to the selection unit 123.
[0120] The second control amount derivation unit 122a of the control amount derivation unit 12a derives the second automatic driving control amount based on a part of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST501 and the second machine learning model 132a. Specifically, the second control amount derivation unit 122a derives the second automatic driving control amount based on the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 in step ST501 and the second machine learning model 132a (step ST503).
[0121] The second control amount derivation unit 122 a outputs the derived second automatic driving control amount to the selection unit 123 .
[0122] The weather determination unit 16 determines the weather around the vehicle 100 based on the information about the current position of the vehicle 100 acquired from the GNSS 23 and the weather information acquired from the cloud weather server 5 (step ST504). The weather determination unit 16 outputs the determined information about the weather around the vehicle 100 to the monitoring unit 14a.
[0123] The monitoring unit 14 a determines whether the reliability of the captured image obtained from the camera 21 has decreased based on the information on the weather output from the weather determination unit 16 in step ST504 .
[0124] Specifically, the monitoring unit 14a determines whether there is fog or precipitation around the vehicle 100, for example, based on the information on the weather output from the weather determination unit 16 (step ST505).
[0125] In step ST505, when it is determined that there is no fog or precipitation around the vehicle 100 (step ST505 is "No"), the monitoring unit 14a determines that the reliability of the captured image acquired from the camera 21 has not decreased, and outputs the monitoring result information indicating that the reliability of the captured image has not decreased to the control unit 15.
[0126] The control unit 15 controls the control amount derivation unit 12a so that the control amount derivation unit 12a outputs the first automatic driving control amount derivation unit 121 in step ST502. The selection unit 123 selects the first automatic driving control amount and outputs it to the vehicle control unit 3 (step ST506).
[0127] In step ST505, when it is determined that there is fog or precipitation around the vehicle 100 (when step ST505 is "yes"), the monitoring unit 14a determines that the reliability of the captured image acquired from the camera 21 is reduced, and outputs monitoring result information indicating that the reliability of the captured image is reduced to the control unit 15.
[0128] The control unit 15 controls the control amount derivation unit 12a so that the control amount derivation unit 12a outputs the second automatic driving control amount derived by the second control amount derivation unit 122a in step ST503. The selection unit 123 selects the second automatic driving control amount and outputs it to the vehicle control unit 3 (step ST507).
[0129] When the action of step ST506 or step ST507 is performed, the action of the automatic driving control device 1a returns to step ST501 and repeats the subsequent actions.
[0130] Regarding the above-described operation, the weather determination unit 16 acquires information related to the current position of the vehicle 100 from the GNSS 23 (see Figure 5 Step ST501), and the weather determination unit 16 determines the weather around the vehicle 100 based on the information related to the current position of the vehicle 100 obtained from the GNSS 23 and the weather information obtained from the cloud weather server 5 (refer to Figure 5 Step ST504 and step ST505 of the above do not need to be performed every time. For example, during the process of step ST501 to step ST507 described above, it can also be performed only once per minute. In the case where the weather determination unit 16 does not perform the process of determining the weather around the vehicle 100 based on the location information and weather information of the vehicle 100 every time, the weather determination unit 16 determines the weather around the vehicle 100 based on the latest weather information obtained from the cloud weather server 5.
[0131] As described above, in a weather condition such as a heavy fog state or a heavy precipitation state where the reliability of the captured image output from the camera 21 is reduced to such an extent that it is not suitable for deriving the automatic driving control amount, the automatic driving control device 1a according to the second embodiment determines that the reliability of the captured image captured by the camera 21 is reduced. When it is determined that the reliability of the captured image is reduced, the automatic driving control device 1a can continue to derive the automatic driving control amount without using the captured image output from the camera 21.
[0132] In addition, in the second embodiment, at least one of the multiple sensors for acquiring the vehicle surrounding information only needs to be the camera 21 , and sensors other than the camera 21 are not limited to the above-mentioned millimeter wave radar 22 .
[0133] However, as a sensor other than the camera 21 , it is necessary to use a sensor whose reliability of the vehicle periphery information outputted from the sensor does not decrease under weather conditions where the reliability of the captured image outputted from the camera 21 decreases.
[0134] In the above-mentioned second embodiment, in the automatic driving control device 1a, the monitoring unit 14a determines whether the reliability of the captured image has decreased (see Figure 5 Before step ST505), the first control amount derivation unit 121 and the second control amount derivation unit 122a respectively derive the first automatic driving control amount and the second automatic driving control amount (refer to Figure 5 Step ST502 and step ST503). However, this is only an example, and the first control amount derivation unit 121 or the second control amount derivation unit 122a can derive the automatic driving control amount by receiving the determination of whether the reliability of the captured image is reduced by the monitoring unit 14a. Specifically, for example, when the monitoring unit 14a determines that the reliability of the captured image is reduced, the control unit 15 controls the control amount derivation unit 12a so that the control amount derivation unit 12a outputs the second automatic driving control amount. When the control amount derivation unit 12a is controlled by the control unit 15 to output the second automatic driving control amount, the second control amount derivation unit 122a derives the second automatic driving control amount. On the other hand, when the monitoring unit 14a determines that the reliability of the captured image is not reduced, the control unit 15 controls the control amount derivation unit 12a so that the control amount derivation unit 12a outputs the first automatic driving control amount. When the control amount deriving unit 12a is controlled by the control unit 15 to output the first automatic driving control amount, the first control amount deriving unit 121 derives the first automatic driving control amount. Based on the control of the control unit 15, the control amount deriving unit 12a outputs the first automatic driving control amount derived by the first control amount deriving unit 121 or the second automatic driving control amount derived by the second control amount deriving unit 122 to the vehicle control unit 3. In the case of such a configuration, the automatic driving control device 1a may not include the selection unit 123.
[0135] As described above, according to Embodiment 2, the automatic driving control device 1a is configured to include: an information acquisition unit 11, which acquires multiple vehicle surrounding information output from multiple sensors (camera 21, millimeter wave radar 22); a control amount derivation unit 12a, which derives the automatic driving control amount based on the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the machine learning model 13a, and outputs the automatic driving control amount; a monitoring unit 14a, which determines whether the reliability of any vehicle surrounding information among the multiple vehicle surrounding information acquired by the information acquisition unit 11 is reduced; and a control unit 15, which controls the control amount derivation unit 12a when the monitoring unit 14a determines that the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, so that the control amount derivation unit 12a outputs the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability.
[0136] Therefore, in the automatic driving control device 1a that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors and the machine learning model 13a, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0137] In more detail, in the automatic driving control device 1a involved in embodiment 2, the control amount derivation unit 12a includes: a first control amount derivation unit 121, which derives a first automatic driving control amount based on all of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the first machine learning model 131; and a second control amount derivation unit 122a, which derives a second automatic driving control amount based on a part of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the second machine learning model 132a. When the monitoring unit 14a determines that the reliability of the vehicle surrounding information among the multiple vehicle surrounding information other than a part of the multiple vehicle surrounding information input into the second machine learning model 132a is reduced, the control unit 15 controls the control amount derivation unit 12a so that the control amount derivation unit 12a outputs the second automatic driving control amount. Therefore, in the automatic driving control device 1a that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors and the machine learning model 13a, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0138] Implementation method 3.
[0139] In the second embodiment, it is assumed that at least one of the multiple sensors is the camera 21. Moreover, as a premise, the possibility of all the multiple sensors failing is low enough to be ignored, while on the other hand, problems are more likely to occur with the captured image output from the camera 21 due to, for example, the influence of weather. Then, the following embodiment is described: the automatic driving control device 1a determines whether the reliability of the captured image output from the camera 21 is reduced based on the weather around the vehicle 100, and even when the reliability of the captured image is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 is derived.
[0140] In the third embodiment, an embodiment will be described in which the automatic driving control device 1 b determines whether the reliability of the captured image output from the camera 21 has decreased by a method different from that of the second embodiment.
[0141] The automatic driving control device 1 b according to the third embodiment is assumed to be mounted on the vehicle 100 similarly to the automatic driving control device 1 according to the first embodiment.
[0142] In the third embodiment, as described below Figure 6 As shown, the sensors are the camera 21 and the millimeter wave radar 22. As described above, in the third embodiment, as a premise, the possibility of failure of the camera 21 and the millimeter wave radar 22 is low enough to be ignored, while on the other hand, problems are more likely to occur with the captured image output from the camera 21.
[0143] Figure 6 This is a diagram showing a configuration example of an automatic driving control device 1b according to the third embodiment.
[0144] Regarding the automatic driving control device 1b according to the third embodiment, Figure 2 The same structures of the automatic driving control device 1 described are marked with the same reference numerals, and repeated descriptions are omitted. In the automatic driving control device 1b involved in the third embodiment, the specific operations of the second control amount derivation unit 122b and the monitoring unit 14b of the control amount derivation unit 12b are different from the specific operations of the second control amount derivation unit 122 and the monitoring unit 14 of the automatic driving control device 1 involved in the first embodiment.
[0145] As described above, in Embodiment 3, as a premise, the captured image output from the camera 21 is more likely to have problems. Therefore, in order to cope with the situation where the reliability of the captured image is reduced, the second machine learning model 132b involved in Embodiment 3 outputs the second automatic driving control amount by taking the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 as input. The second control amount derivation unit 122b derives the second automatic driving control amount based on the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 and the second machine learning model 132b.
[0146] The monitoring unit 14 b determines whether the reliability of the captured image has decreased based on the brightness of the captured image acquired by the information acquisition unit 11 .
[0147] Specifically, for example, when the maximum value of the brightness of each pixel of the captured image acquired by the information acquisition unit 11 is below a preset threshold value (hereinafter referred to as a "brightness determination threshold value"), the monitoring unit 14b determines that the reliability of the captured image obtained from the camera 21 is reduced. In the brightness determination threshold value, for example, a brightness value is set in advance so that when the brightness of all pixels of the captured image is below the brightness determination threshold value, the entire captured image becomes an image that is so dark that an object cannot be recognized.
[0148] Here, FIG. 7 is a diagram for explaining an example of a captured image in which the brightness of pixels is different in the third embodiment. Fig. 7A is an example of a captured image in which all pixels of the captured image have brightness sufficient to recognize an object in the captured image, Figure 7B is an example of a captured image in which all pixels of the captured image have brightness such that the image is so dark that an object in the captured image cannot be recognized, Figure 7C This is an example of a captured image in which all pixels of the captured image have brightness that makes the image so bright that an object in the captured image cannot be recognized.
[0149] For example, in a captured image, when the brightness of all black pixels is set to “0” and the brightness of all white pixels is set to “255”, the brightness determination threshold is set to “5”.
[0150] In addition, if Figure 7C As shown, even if the captured image is too bright, the object in the captured image cannot be recognized. Therefore, for example, in the brightness determination threshold, the brightness value can be pre-set so that when the brightness of all pixels of the captured image is above the brightness determination threshold, the entire captured image becomes an image that is so bright that the object cannot be recognized. At this time, when the minimum value of the brightness of each pixel of the captured image acquired by the information acquisition unit 11 is above the brightness determination threshold, the monitoring unit 14b determines that the reliability of the captured image obtained from the camera 21 is reduced. For example, in the captured image, when the brightness of all black pixels is set to "0" and the brightness of all white pixels is set to "255", the brightness determination threshold is set to "250".
[0151] The operation of the automatic driving control device 1 b according to the third embodiment will be described.
[0152] Figure 8 This is a flowchart for explaining the operation of the automatic driving control device 1b according to the third embodiment.
[0153] The information acquisition unit 11 acquires a plurality of vehicle surrounding information outputted from a plurality of sensors. Specifically, the information acquisition unit 11 acquires the captured image captured by the camera 21 and the distance information measured by the millimeter wave radar 22 as the vehicle surrounding information (step ST801). The information acquisition unit 11 outputs the acquired vehicle surrounding information to the control amount derivation unit 12b and the monitoring unit 14b.
[0154] The first control amount derivation unit 121 of the control amount derivation unit 12b derives the first automatic driving control amount based on all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST801 and the first machine learning model 131. Specifically, the first control amount derivation unit 121 derives the first automatic driving control amount based on the captured image acquired by the information acquisition unit 11 from the camera 21 in step ST801, the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 in step ST801, and the first machine learning model 131 (step ST802). The first control amount derivation unit 121 outputs the derived first automatic driving control amount to the selection unit 123.
[0155] The second control amount derivation unit 122b of the control amount derivation unit 12 derives the second automatic driving control amount based on a part of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST801 and the second machine learning model 132b. Specifically, the second control amount derivation unit 122b derives the second automatic driving control amount based on the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 in step ST801 and the second machine learning model 132b (step ST803). The second control amount derivation unit 122b outputs the derived second automatic driving control amount to the selection unit 123.
[0156] The monitoring unit 14b determines whether the reliability of any vehicle surrounding information among the plurality of vehicle surrounding information acquired by the information acquisition unit 11 has decreased. Specifically, first, the monitoring unit 14b causes the information acquisition unit 11 to acquire a captured image acquired from the camera 21 in step ST801 (step ST804).
[0157] The monitoring unit 14 b determines whether the maximum value of the brightness of each pixel of the captured image acquired in step ST804 is equal to or less than the brightness determination threshold value (step ST805 ).
[0158] In step ST805, when it is determined that the maximum value of the brightness of the pixels in the captured image is greater than the brightness determination threshold (when step ST805 is "No"), the monitoring unit 14b determines that the reliability of the captured image acquired from the camera 21 has not decreased, and outputs the monitoring result information indicating that the reliability of the captured image has not decreased to the control unit 15.
[0159] The control unit 15 controls the control amount derivation unit 12b so that the control amount derivation unit 12b outputs the first automatic driving control amount derived by the first control amount derivation unit 121 in step ST802. The selection unit 123 selects the first automatic driving control amount and outputs it to the vehicle control unit 3 (step ST806).
[0160] In step ST805, when it is determined that the maximum value of the brightness of the pixels in the image is below the brightness determination threshold (when step ST805 is "yes"), the monitoring unit 14b determines that the reliability of the captured image acquired from the camera 21 is reduced, and outputs the monitoring result information indicating that the reliability of the captured image is reduced to the control unit 15.
[0161] The control unit 15 controls the control amount derivation unit 12b so that the control amount derivation unit 12b outputs the second automatic driving control amount derived by the second control amount derivation unit 122b in step ST803. The selection unit 123 selects the second automatic driving control amount and outputs it to the vehicle control unit 3 (step ST807).
[0162] When the action of step ST806 or step ST807 is performed, the action of the automatic driving control device 1b returns to step ST801 and repeats the subsequent actions.
[0163] As described above, when the captured image output from the camera 21 is a captured image with a brightness that makes it impossible to identify an object in the captured image, such that the reliability of the captured image is reduced to the point that it is not suitable for the derivation of the automatic driving control amount, the automatic driving control device 1b involved in the third embodiment determines that the reliability of the captured image captured by the camera 21 is reduced. In the case where it is determined that the reliability of the captured image is reduced, the automatic driving control device 1b can continue to implement the derivation of the automatic driving control amount without using the captured image. The captured image with a brightness that makes it impossible to identify an object in the captured image refers to a captured image obtained, for example, in a dark condition, in a condition where the exposure correction function of the camera 21 is poor, or in a condition where there is an obstruction in front of the camera 21 and the camera cannot be photographed. In this case, the captured image obtained by the automatic driving control device 1b from the camera 21 is, for example, an image that is so dark that the object in the captured image cannot be identified, or an image that is so bright that the object in the captured image cannot be identified.
[0164] In addition, in the third embodiment, at least one of the multiple sensors for acquiring the vehicle surrounding information only needs to be the camera 21 , and sensors other than the camera 21 are not limited to the above-mentioned millimeter wave radar 22 .
[0165] As described above, according to embodiment 3, the automatic driving control device 1b is configured to include: an information acquisition unit 11, which acquires multiple vehicle surrounding information output from multiple sensors (camera 21, millimeter wave radar 22); a control amount derivation unit 12b, which derives the automatic driving control amount based on the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the machine learning model 13b, and outputs the automatic driving control amount; a monitoring unit 14b, which determines whether the reliability of any vehicle surrounding information among the multiple vehicle surrounding information acquired by the information acquisition unit 11 is reduced; and a control unit 15, which controls the control amount derivation unit 12b when the monitoring unit 14b determines that the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, so that the control amount derivation unit 12b outputs the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability.
[0166] Therefore, in the automatic driving control device 1b that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors respectively and the machine learning model 13b, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0167] In more detail, in the automatic driving control device 1b involved in embodiment 3, the control amount derivation unit 12b includes: a first control amount derivation unit 121, which derivates the first automatic driving control amount based on all of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the first machine learning model 131; and a second control amount derivation unit 122b, which derivates the second automatic driving control amount based on a part of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the second machine learning model 132b. When the monitoring unit 14b determines that the reliability of the vehicle surrounding information among the multiple vehicle surrounding information other than a part of the multiple vehicle surrounding information input into the second machine learning model 132b is reduced, the control unit 15 controls the control amount derivation unit 12b so that the control amount derivation unit 12b outputs the second automatic driving control amount. Therefore, in the automatic driving control device 1b that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors respectively and the machine learning model 13b, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0168] Implementation method 4.
[0169] In Embodiments 2 and 3, it is assumed as a premise that at least one of the multiple sensors is the camera 21, and the possibility of failure of each of the multiple sensors is low enough to be ignored, while on the other hand, the captured image output from the camera 21 is more likely to have problems. Then, the following embodiment is described: the automatic driving control device 1a, 1b determines whether the reliability of the captured image output from the camera 21 is reduced based on the weather around the vehicle 100 or the brightness of the pixels of the captured image, and even if the reliability of the captured image is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 is derived.
[0170] In the fourth embodiment, an embodiment will be described in which the automatic driving control device 1 c determines whether the reliability of the captured image output from the camera 21 has decreased by a method different from that of the second and third embodiments.
[0171] The automatic driving control device 1 c according to the fourth embodiment is assumed to be mounted on the vehicle 100 similarly to the automatic driving control device 1 according to the first to third embodiments.
[0172] In the fourth embodiment, as described below Fig. 9 As shown, the sensors are the camera 21 and the millimeter wave radar 22. Embodiment 4 is also the same as Embodiment 2 and Embodiment 3, as a premise, the possibility of failure of the camera 21 and the millimeter wave radar 22 is low to a negligible level, while on the other hand, problems are more likely to occur with the captured image output from the camera 21.
[0173] The automatic driving control device 1c, similarly to Embodiments 1 to 3, obtains the captured image output from the camera 21 and the distance information output from the millimeter wave radar 22 as vehicle surrounding information. In addition, in Embodiment 4, the automatic driving control device 1c outputs information from the vehicle driving sensor 24 for determining whether the vehicle 100 is driving (hereinafter referred to as "vehicle driving information"). In Embodiment 4, "the vehicle 100 is driving" is synonymous with "the vehicle 100 is moving". "The vehicle 100 is moving" is synonymous with "the speed of the vehicle 100 is not '0'".
[0174] The vehicle driving sensor 24 outputs vehicle driving information. The vehicle driving sensor 24 may output information that can determine whether the vehicle 100 is driving. For example, the vehicle driving sensor 24 may be a sensor that obtains the number of rotations of the wheels, or a GNSS that obtains information related to the current position of the vehicle 100.
[0175] Furthermore, the travel determination unit 17 described later determines whether the vehicle 100 is traveling based on vehicle travel information output from the vehicle travel sensor 24 .
[0176] Fig. 9 This is a diagram showing a configuration example of an automatic driving control device 1c according to a fourth embodiment.
[0177] Regarding the automatic driving control device 1c according to the fourth embodiment, Figure 2 The same structures as the described automatic driving control device 1 are marked with the same reference numerals, and repeated descriptions are omitted. The structure of the automatic driving control device 1c involved in the fourth embodiment is different from the structure of the automatic driving control device 1 involved in the first embodiment in that it includes a driving determination unit 17. In addition, in the automatic driving control device 1c involved in the fourth embodiment, the specific operations of the second control amount derivation unit 122c of the control amount derivation unit 12c and the monitoring unit 14c are different from the specific operations of the second control amount derivation unit 122 and the monitoring unit 14 of the automatic driving control device 1 involved in the first embodiment.
[0178] As described above, in Embodiment 4, as a premise, the captured image output from the camera 21 is more likely to have problems. Therefore, in order to cope with the situation where the reliability of the captured image is reduced, the second machine learning model 132c involved in Embodiment 4 outputs the second automatic driving control amount by taking the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 as input. The second control amount derivation unit 122c derives the second automatic driving control amount based on the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 and the second machine learning model 132c.
[0179] The information acquisition unit 11 acquires, as vehicle surrounding information, the image captured by the camera 21 and the distance information measured by the millimeter wave radar 22. The information acquisition unit 11 outputs the acquired vehicle surrounding information to the control amount derivation unit 12c and the monitoring unit 14c.
[0180] The travel determination unit 17 determines whether the vehicle 100 is traveling based on the vehicle travel information acquired from the vehicle travel sensor 24 .
[0181] The traveling determination unit 17 outputs information on whether the determined vehicle 100 is traveling to the monitoring unit 14 c .
[0182] The monitoring unit 14c determines whether the reliability of the captured image has decreased based on the information on whether the vehicle 100 is traveling acquired from the traveling determination unit 17 and the captured image acquired from the information acquisition unit 11. Specifically, the monitoring unit 14c determines whether the vehicle 100 is traveling and whether the scenery around the vehicle 100 captured in the captured image acquired from the camera 21 has not changed. The monitoring unit 14c determines whether the scenery captured in the captured image has not changed based on the captured image acquired from the information acquisition unit 11 and the captured image stored in the storage unit. For example, the monitoring unit 14c compares the captured image acquired from the information acquisition unit 11 (referred to as "the first captured image") and the captured image most recently stored in the storage unit (referred to as "the second captured image") in units of pixels, and when the comparison result satisfies a predetermined comparison condition, it is determined that the scenery captured in the first captured image has not changed. The predetermined comparison condition is, for example, that the average of the absolute value of the difference between the pixel values of each pixel of the first captured image and the second captured image is less than a predetermined threshold value. This is merely an example, and the comparison condition may be set to a condition that can determine that the first captured image and the second captured image are identical. In Embodiment 4, the first captured image and the second captured image are identical, but are not limited to being completely identical, and may also be substantially identical.
[0183] When the vehicle 100 is traveling and the scenery around the vehicle 100 captured in the captured image has not changed, the monitoring unit 14 c determines that the reliability of the captured image has decreased.
[0184] It is assumed that the scenery captured in the captured image changes while the vehicle 100 is traveling. Therefore, whether the vehicle 100 is traveling or not, the fact that the scenery captured in the captured image does not change means that the reliability of the captured image is reduced.
[0185] In the fourth embodiment, in the automatic driving control device 1c, the information acquisition unit 11 stores the captured image acquired from the camera 21 in the storage unit. The monitoring unit 14c determines whether the scenery captured in the captured image has not changed based on the captured image acquired from the information acquisition unit 11 and the captured image stored in the storage unit.
[0186] The operation of the automatic driving control device 1 c according to the fourth embodiment will be described.
[0187] Fig.10 This is a flowchart for explaining the operation of the automatic driving control device 1c involved in the fourth embodiment.
[0188] The information acquisition unit 11 acquires a plurality of vehicle surrounding information outputted from a plurality of sensors. Specifically, the information acquisition unit 11 acquires the captured image captured by the camera 21 and the distance information measured by the millimeter wave radar 22 as the vehicle surrounding information. In addition, the driving determination unit 17 acquires the vehicle driving information from the vehicle driving sensor 24 (step ST1001). The information acquisition unit 11 outputs the acquired vehicle surrounding information to the control amount derivation unit 12c and the monitoring unit 14c.
[0189] The first control amount derivation unit 121 of the control amount derivation unit 12c derives the first automatic driving control amount based on all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST1001 and the first machine learning model 131. Specifically, the first control amount derivation unit 121 derives the first automatic driving control amount based on the captured image acquired by the information acquisition unit 11 from the camera 21 in step ST1001, the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 in step ST1001, and the first machine learning model 131 (step ST1002). The first control amount derivation unit 121 outputs the derived first automatic driving control amount to the selection unit 123.
[0190] The second control amount derivation unit 122c of the control amount derivation unit 12c derives the second automatic driving control amount based on a part of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 in step ST1001 and the second machine learning model 132c. Specifically, the second control amount derivation unit 122c derives the second automatic driving control amount based on the distance information acquired by the information acquisition unit 11 from the millimeter wave radar 22 in step ST1001 and the second machine learning model 132c (step ST1003).
[0191] The second control amount derivation unit 122 c outputs the derived second automatic driving control amount to the selection unit 123 .
[0192] The travel determination unit 17 determines whether the vehicle 100 is traveling based on the vehicle travel information acquired from the vehicle travel sensor 24 (step ST1004). The travel determination unit 17 outputs information on whether the vehicle 100 is traveling to the monitoring unit 14c.
[0193] The monitoring unit 14c determines whether a captured image acquired from the camera 21 in the past is stored in the storage unit (step ST1005).
[0194] In step ST1005, when it is determined that the captured image is stored in the storage unit (step ST1005 is "No"), the monitoring unit 14c determines that the reliability of the captured image acquired from the camera 21 has not decreased, and outputs monitoring result information indicating that the reliability of the captured image has not decreased to the control unit 15. The operation of the automatic driving control device 1c proceeds to step ST1007.
[0195] In step ST1005, when it is determined that the captured image is stored in the storage unit (if step ST1005 is "yes"), the monitoring unit 14c determines whether the reliability of the captured image has decreased based on the captured image output from the information acquisition unit 11 in step ST1001 and the information on whether the vehicle 100 is traveling output from the travel determination unit 17 in step ST1004. Specifically, the monitoring unit 14c determines whether the vehicle 100 is traveling and whether the scenery around the vehicle 100 captured in the captured image has not changed.
[0196] In step ST1006, when the monitoring unit 14c determines that the vehicle 100 is not traveling, or determines that the scenery around the vehicle 100 captured in the captured image has changed (if step ST1006 is "No"), the monitoring unit 14c determines that the reliability of the captured image acquired from the camera 21 has not decreased, and outputs monitoring result information indicating that the reliability of the captured image has not decreased to the control unit 15. The operation of the automatic driving control device 1c proceeds to step ST1007.
[0197] In step ST1007, the control unit 15 controls the control amount derivation unit 12c so that the control amount derivation unit 12c outputs the first automatic driving control amount derivation unit 121 in step ST1002. The selection unit 123 selects the first automatic driving control amount and outputs it to the vehicle control unit 3 (step ST1007).
[0198] On the other hand, in step ST1006, when it is determined that the vehicle 100 is traveling and the scenery around the vehicle 100 captured in the captured image has not changed (if step ST1006 is "yes"), the monitoring unit 14c determines that the reliability of the captured image acquired from the camera 21 is reduced, and outputs monitoring result information indicating that the reliability of the captured image is reduced to the control unit 15.
[0199] The control unit 15 controls the control amount derivation unit 12c so that the control amount derivation unit 12c outputs the second automatic driving control amount derivation unit 122c in step ST1003. The selection unit 123 selects the second automatic driving control amount and outputs it to the vehicle control unit 3 (step ST1008).
[0200] When the action of step ST1007 or step ST1008 is performed, the action of the automatic driving control device 1c returns to step ST1001 and repeats the subsequent actions.
[0201] As described above, the automatic driving control device 1c according to the fourth embodiment determines that the reliability of the photographic image acquired from the camera 21 is reduced, when a situation such as the automatic driving control amount cannot be correctly derived or only a photographic image that cannot properly capture the vehicle surrounding conditions in the traveling vehicle 100 is acquired. When it is determined that the reliability of the photographic image is reduced, the automatic driving control device 1c can continue to derive the automatic driving control amount without using the photographic image output from the camera 21.
[0202] In addition, in the fourth embodiment, at least one of the multiple sensors for acquiring the vehicle surrounding information only needs to be the camera 21 , and sensors other than the camera 21 are not limited to the above-mentioned millimeter wave radar 22 .
[0203] As described above, according to Embodiment 4, the automatic driving control device 1c is configured to include: an information acquisition unit 11, which acquires multiple vehicle surrounding information output from multiple sensors (camera 21, millimeter wave radar 22); a control amount derivation unit 12c, which derives the automatic driving control amount based on the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the machine learning model 13c, and outputs the automatic driving control amount; a monitoring unit 14c, which determines whether the reliability of any vehicle surrounding information among the multiple vehicle surrounding information acquired by the information acquisition unit 11 is reduced; and a control unit 15, which, when the monitoring unit 14c determines that the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, controls the control amount derivation unit 12c so that the control amount derivation unit 12c outputs the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability.
[0204] Therefore, in the automatic driving control device 1c that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors and the machine learning model 13c, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0205] In more detail, in the automatic driving control device 1c involved in embodiment 4, the control amount derivation unit 12c includes: a first control amount derivation unit 121, which derivates the first automatic driving control amount based on all of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the first machine learning model 131; and a second control amount derivation unit 122c, which derivates the second automatic driving control amount based on a part of the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the second machine learning model 132c. When the monitoring unit 14c determines that the reliability of the vehicle surrounding information among the multiple vehicle surrounding information other than a part of the multiple vehicle surrounding information input into the second machine learning model 132c is reduced, the control unit 15 controls the control amount derivation unit 12c so that the control amount derivation unit 12c outputs the second automatic driving control amount. Therefore, in the automatic driving control device 1c that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors and the machine learning model 13c, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0206] Implementation method 5.
[0207] In the automatic driving control device 1 according to the first embodiment, when it is determined that the reliability of the vehicle surrounding information output from the sensor has decreased, information indicating the decrease in reliability can be output to the outside of the automatic driving control device 1 .
[0208] In the fifth embodiment, an embodiment will be described in which, when the automatic driving control device 1 determines that the reliability of the vehicle surrounding information output from the sensor has decreased, the automatic driving control device 1 can output information indicating the decrease in reliability to the outside of the automatic driving control device 1 .
[0209] Fig.11 This is a diagram showing a configuration example of an automatic driving control device 1d according to the fifth embodiment.
[0210] Regarding the automatic driving control device 1d according to the fifth embodiment, the automatic driving control device 1d according to the first embodiment Figure 2 The same components of the automatic driving control device 1 described above are denoted by the same reference numerals, and repeated descriptions are omitted. The structure of the automatic driving control device 1 d according to the fifth embodiment differs from the structure of the automatic driving control device 1 according to the first embodiment in that it includes a notification control unit 18 .
[0211] When the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount, the notification control unit 18 outputs notification information indicating that the reliability of the vehicle surrounding information of a part of the plurality of vehicle surrounding information is reduced. Specifically, when the monitoring unit 14 determines that the reliability of the distance information is reduced and the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount, the notification control unit 18 outputs notification information indicating that the reliability of the distance information output from the millimeter wave radar 22 is reduced. The notification control unit 18 outputs the notification information to an output device (not shown) connected to the automatic driving control device 1d via a network. The output device is, for example, a display provided by a car navigation system mounted on the vehicle 100. The notification control unit 18 causes the display to display the notification information.
[0212] In embodiment 5, when the monitoring unit 14 determines that the reliability of vehicle surrounding information other than a part of the multiple vehicle surrounding information is reduced, the control unit 15 controls to output a second automatic driving control amount, and outputs information indicating that the reliability of the vehicle surrounding information is reduced to the notification control unit 18.
[0213] here, Fig.12 and Fig.13 This is a diagram showing an example of an image of a display on which the notification control unit 18 displays notification information in the fifth embodiment.
[0214] For example, Fig.12 As shown, the notification control unit 18 causes the display to display a message "Millimeter wave radar cannot be used" as information indicating that the reliability of the distance information output from the millimeter wave radar 22 has decreased (see Fig.12 1201) As described above, the information indicating that the reliability of the distance information is reduced includes, for example, a message indicating that a sensor that outputs vehicle surrounding information with reduced reliability is unusable.
[0215] In addition, for example, Fig.13 As shown, the notification control unit 18 causes the display to display a message "The lane change function is currently unavailable" as information indicating that the reliability of the distance information output from the millimeter wave radar 22 has decreased (see Fig.13 As described above, the information indicating that the reliability of the distance information is reduced includes, for example, a message notifying that the function of the automatic driving control of the vehicle 100 cannot be used because there is vehicle surrounding information determined to have reduced reliability. For example, when the distance information output from the millimeter wave radar 22 cannot be used in the automatic driving control of the vehicle 100, lane change cannot be performed.
[0216] The operation of the automatic driving control device 1d according to the fifth embodiment will be described.
[0217] The operation of the automatic driving control device 1d is similar to that used in the first embodiment. Figure 3 The operations to be described are basically the same, so the illustration of a flowchart is omitted.
[0218] In step ST305, when it is determined that the difference between the reference distance acquired in step ST304 and the distance from the vehicle 100 to the object based on the distance information acquired by the information acquisition unit 11 from the millimeter-wave radar 22 in step ST301 is greater than the radar determination threshold (the answer is "No" in step ST305), the monitoring unit 14 determines that the reliability of the distance information acquired from the millimeter-wave radar 22 is reduced, and outputs the monitoring result information indicating that the reliability of the distance information is reduced to the control unit 15.
[0219] The control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount derivation unit 122 in step ST303, and outputs information indicating that the reliability of the vehicle surrounding information is reduced to the notification control unit 18. The notification control unit 18 outputs notification information indicating that the reliability of a part of the vehicle surrounding information among the plurality of vehicle surrounding information is reduced.
[0220] As described above, in the automatic driving control device 1d involved in the fifth embodiment, when the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount, the notification control unit 18 outputs notification information indicating that the reliability of a part of the vehicle surrounding information among the plurality of vehicle surrounding information has decreased. Thus, the automatic driving control device 1d can notify the driver of the vehicle 100 or the like of the fact that the reliability of a part of the vehicle surrounding information has decreased. When the driver or the like confirms that the reliability of a part of the vehicle surrounding information has decreased, for example, the driver or the like confirms whether the antenna of the millimeter wave radar 22 is not dirty, and if the antenna is dirty, the antenna is cleaned. Alternatively, the driver or the like repairs the millimeter wave radar 22.
[0221] In the automatic driving control device 1d according to the fifth embodiment, the notification control unit 18 outputs, for example, Fig.13 In the case of the notification information shown in FIG. 1 , the driver or the like can be notified that the function of the autonomous driving that can be derived has changed. As a result, the autonomous driving control device 1d can make the driver or the like understand that the desired autonomous driving function cannot be used, thereby preventing the driver or the like from being confused about the fact that the desired autonomous driving function cannot be used. As described above, although the notification control unit 18 can also output the following information: Fig.12 The notification information shown, but Fig.13 As shown, the function reduction in the automatic driving control is notified, so that the driver and the like can specifically understand which function of the automatic driving cannot be used.
[0222] In the above embodiment 5, as an example, the output device for the notification control unit 18 to output the notification information is a display provided by the car navigation system, but this is only an example. For example, the output device for the notification control unit 18 to output the notification information may be a dashboard, and the notification control unit 18 may display the notification information in the form of a message or an icon. In addition, the notification control unit 18 is not limited to displaying the notification information, but may also output the notification information by sound. Specifically, for example, the output device may be a sound output device such as a speaker, and the notification control unit 18 may output the notification information from the sound output device. The notification control unit 18 may output the notification information in the form of an automatic sound, or may simply output the notification information in the form of a buzzer sound. In addition, the notification control unit 18 may display the notification information as a message on the display, and output it as a sound or a buzzer sound.
[0223] In addition, the structure of the automatic driving control device 1d involved in the above-described embodiment 5 can also be applied to the above-described embodiments 2 to 4. That is, in the automatic driving control device 1a involved in embodiment 2, the automatic driving control device 1b involved in embodiment 3, or the automatic driving control device 1c involved in embodiment 4, a notification control unit 18 is provided, and the notification control unit 18 can also output information indicating that the reliability of the captured image obtained from the camera 21 is reduced.
[0224] As described above, according to the fifth embodiment, the automatic driving control device 1d includes, in addition to the structure of the automatic driving control device 1 involved in the first embodiment, the notification control unit 18, which outputs notification information indicating that the reliability of the vehicle surrounding information of a part of the plurality of vehicle surrounding information has decreased when the control unit 15 controls the control amount derivation unit 12 so that the control amount derivation unit 12 outputs the second automatic driving control amount. Therefore, when the automatic driving control device 1d determines that the reliability of the vehicle surrounding information output from the sensor has decreased, the automatic driving control device 1d can notify the driver or the like of the decrease in reliability.
[0225] Implementation method 6.
[0226] The automatic driving control devices 1 to 1d involved in embodiments 1 to 5 use the first machine learning model 131 when there is no vehicle surrounding information with reduced reliability among the multiple vehicle surrounding information output from the multiple sensors, and use the second machine learning models 132, 132a to 132c when there is vehicle surrounding information with reduced reliability among the multiple vehicle surrounding information output from the multiple sensors.
[0227] However, these are merely examples, and the autonomous driving control device can use the same machine learning model 13 for multiple vehicle surrounding information output from multiple sensors, regardless of whether there is vehicle surrounding information with reduced reliability or vehicle surrounding information with reduced reliability. In Implementation 6, an implementation method in which the autonomous driving control device uses the same machine learning model in either of the above two situations is described.
[0228] Fig.14 This is a diagram showing a configuration example of an automatic driving control device 1e according to the sixth embodiment.
[0229] In addition, here, as an example, the structure and operation of the automatic driving control device 1e involved in the sixth embodiment are described as an example in which a part of the structure and operation of the automatic driving control device 1 involved in the first embodiment is changed. However, the structure and operation of the automatic driving control device 1e involved in the sixth embodiment can also be realized by changing a part of the structure and operation of any of the automatic driving control devices 1a to 1d involved in the second to fifth embodiments.
[0230] Regarding the automatic driving control device 1e according to the sixth embodiment, Figure 2 The same structures as those of the described automatic driving control device 1 are marked with the same reference numerals, and repeated descriptions are omitted. The structure of the automatic driving control device 1e involved in Embodiment 6 is different from the structure of the automatic driving control device 1 involved in Embodiment 1 in that the control amount derivation unit 12e does not include the first control amount derivation unit 121, the second control amount derivation unit 122 and the selection unit 123, and the machine learning model 13e does not include the first machine learning model 131 and the second machine learning model 132.
[0231] The automatic driving control device 1e according to the sixth embodiment differs from the automatic driving control device 1 according to the first embodiment in the operation of the control unit 15e.
[0232] The control unit 15e gives an information validity flag obtained based on the reliability determination result performed by the monitoring unit 14 to all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11. The information validity flag is information indicating whether each vehicle surrounding information is valid or invalid. That is, when the reliability of a certain vehicle surrounding information is not reduced, the information validity flag indicating that the vehicle surrounding information is valid is given. In addition, when the reliability of a certain vehicle surrounding information is reduced, the information validity flag indicating that the vehicle surrounding information is invalid is given.
[0233] In embodiment 6, as described below Fig.14As shown, the sensors are the camera 21 and the millimeter wave radar 22, similarly to the sensors in Embodiment 1. In Embodiment 6, similarly to the premise of Embodiment 1, the image captured by the camera 21 does not substantially cause a problem, but on the other hand, the distance information of the millimeter wave radar 22 is more likely to cause a problem.
[0234] Therefore, specifically, when the monitoring unit 14 determines that the reliability of the distance information output from the millimeter wave radar 22 is reduced, the control unit 15e assigns an information validity flag "1" to the captured image output from the camera 21, and assigns an information validity flag "0" to the distance information output from the millimeter wave radar 22. In the sixth embodiment, as an example, when the information validity flag is "1", it indicates that the reliability of the vehicle surrounding information assigned with the information validity flag is not reduced, and when the information validity flag is "0", it indicates that the reliability of the marked vehicle surrounding information assigned with the information validity flag is reduced.
[0235] Hereinafter, the vehicle surrounding information to which the information validity flag is assigned is referred to as “flagged vehicle surrounding information”.
[0236] The control unit 15 e outputs the vehicle surrounding information with a plurality of flags generated by assigning information validity flags to all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 to the control amount derivation unit 12 e.
[0237] When multiple vehicle surrounding information with signs is obtained from the control unit 15e, the control amount derivation unit 12e derives the automatic driving control amount based on all of the multiple vehicle surrounding information with signs and the machine learning model 13e, and outputs the automatic driving control amount.
[0238] The machine learning model 13e takes all of the multiple marked vehicle surrounding information output from the control unit 15e as input and outputs the autonomous driving control amount. The machine learning model 13e is a model that learns in a manner that can derive the autonomous driving control amount by removing the influence of the vehicle surrounding information that is assigned "0" as the information validity flag from the multiple marked vehicle surrounding information. For example, such learning can be performed based on learning data that is obtained by combining the multiple marked vehicle surrounding information that can be input to the machine learning model 13e and the correct answer of the ideal autonomous driving control amount that is pre-derived based only on the valid vehicle surrounding information among the multiple marked vehicle surrounding information.
[0239] The control amount derivation unit 12e derives the automatic driving control amount based on the machine learning model 13e and the plurality of marked vehicle surrounding information, thereby being able to derive the automatic driving control amount obtained after removing the influence of the vehicle surrounding information with reduced reliability, and output the automatic driving control amount. Here, the so-called "removing the influence of the vehicle surrounding information with reduced reliability" includes not only a state in which the influence of the vehicle surrounding information with reduced reliability is completely removed, but also a state in which the influence of the vehicle surrounding information with reduced reliability is substantially removed to the extent that the automatic driving control amount can be obtained to continue the automatic driving control.
[0240] The operation of the automatic driving control device 1 e according to the sixth embodiment will be described.
[0241] Fig.15 This is a flowchart for explaining the operation of the automatic driving control device 1e involved in the sixth embodiment.
[0242] Fig.15 The specific actions of step ST1501 to step ST1502 are respectively the same as those described in Implementation Example 1. Figure 3 The specific actions of step ST301 and step ST304 are the same, so repeated descriptions are omitted.
[0243] The monitoring unit 14 performs a determination process to determine whether the reliability of the distance information obtained from the millimeter wave radar 22 has decreased (step ST1503). Specifically, the monitoring unit 14 calculates the difference between the reference distance obtained in step ST1502 and the distance from the vehicle 100 to the object obtained based on the distance information obtained from the millimeter wave radar 22 by the information acquisition unit 11 in step ST1501, and determines whether the calculated difference is below the radar determination threshold (step ST1503). The specific operation of step ST1503 is the same as that described in Implementation Example 1. Figure 3 The specific actions of step ST305 are the same.
[0244] When it is determined that the calculated difference is greater than the radar determination threshold, the monitoring unit 14 determines that the reliability of the distance information acquired from the millimeter wave radar 22 is reduced, and outputs monitoring result information indicating that the reliability of the distance information is reduced to the control unit 15e.
[0245] On the other hand, when it is determined that the calculated difference is below the radar judgment threshold, the monitoring unit 14 determines that the reliability of the distance information obtained from the millimeter wave radar 22 has not decreased, and outputs monitoring result information indicating that the reliability of the distance information has not decreased to the control unit 15e.
[0246] The control unit 15e assigns an information validity flag to the vehicle surrounding information based on the determination result determined by the monitoring unit 14 in step ST1503 (step ST1504). Specifically, for example, when the monitoring unit 14 determines that the reliability of the distance information obtained from the millimeter wave radar 22 is reduced, the control unit 15e assigns an information validity flag "0" to the distance information. The control unit 15e can obtain the distance information from the monitoring unit 14. For example, when the monitoring unit 14 determines that the reliability of the distance information obtained from the millimeter wave radar 22 is not reduced, the control unit 15e assigns an information validity flag "1" to the distance information.
[0247] In the sixth embodiment, similarly to the premise of the first embodiment, there is substantially no problem with the image captured by the camera 21, and therefore the control unit 15e always gives the information validity flag "1" to the captured image.
[0248] The control unit 15e outputs the marked vehicle surrounding information to the control amount derivation unit 12e.
[0249] The control amount derivation unit 12e derives the automatic driving control amount of the vehicle 100 based on the marked vehicle surrounding information output from the control unit 15e and the machine learning model 13e (step ST1505). Then, the control amount derivation unit 12e outputs the vehicle control information based on the derived automatic driving control amount to the vehicle control unit 3 (step ST1506).
[0250] As described above, the automatic driving control device 1e may include only one control amount derivation unit 12e and a machine learning model 13e. Thus, it is not necessary to prepare a plurality of machine learning models corresponding to the number of vehicle peripheral information input, and the automatic driving control amount can be deduced with a simpler structure than when a plurality of machine learning models corresponding to the number of vehicle peripheral information are prepared in advance.
[0251] In addition, when deriving the automatic driving control amount, the automatic driving control device 1e gives the vehicle surrounding information an information validity flag, thereby being able to determine whether the vehicle surrounding information is valid for deriving the automatic driving control amount. Therefore, even if the reliability of the vehicle surrounding information output from one or more sensors among the plurality of sensors is reduced, the automatic driving control device 1e can derive the automatic control amount that can be used for the automatic driving control of the vehicle 100.
[0252] As described above, according to embodiment 6, the automatic driving control device 1e is configured to include: an information acquisition unit 11, which acquires multiple vehicle surrounding information output from multiple sensors (camera 21, millimeter wave radar 22); a control amount derivation unit 12e, which derives the automatic driving control amount based on the multiple vehicle surrounding information acquired by the information acquisition unit 11 and the machine learning model 13e, and outputs the automatic driving control amount; a monitoring unit 14, which determines whether the reliability of any vehicle surrounding information among the multiple vehicle surrounding information acquired by the information acquisition unit 11 is reduced; and a control unit 15e, which controls the control amount derivation unit 12e when the monitoring unit 14 determines that the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, so that the control amount derivation unit 12e outputs the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability.
[0253] Therefore, in the automatic driving control device 1e that derives and outputs the automatic driving control amount based on multiple vehicle surrounding information output from multiple sensors and the machine learning model 13e, even if the reliability of any vehicle surrounding information among the multiple vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output.
[0254] In more detail, in the automatic driving control device 1e involved in the sixth embodiment, the control unit 15e assigns an information validity flag to all of the plurality of vehicle surrounding information acquired by the information acquisition unit 11 based on the reliability determination result performed by the monitoring unit 14, and the control amount derivation unit 12e derivates the automatic driving control amount based on all of the plurality of vehicle surrounding information assigned the information validity flag by the control unit 15e and the machine learning model 13e, and outputs the automatic driving control amount. Therefore, in the automatic driving control device 1e that derivates and outputs the automatic driving control amount based on the plurality of vehicle surrounding information outputted from the plurality of sensors and the machine learning model 13e, even when the reliability of any of the plurality of vehicle surrounding information is reduced, the automatic driving control amount suitable for the automatic driving control of the vehicle 100 can be output. In addition, compared with the case where the machine learning model 13e is prepared in advance according to the type of vehicle surrounding information, the automatic driving control amount can be deduced with a simpler structure.
[0255] Fig.16A , Fig. 16B This is a diagram showing an example of the hardware configuration of automatic driving control devices 1 to 1 e according to Embodiments 1 to 6.
[0256] In the first to sixth embodiments, the functions of the information acquisition unit 11, the control amount derivation units 12 to 12e, the monitoring units 14 to 14c, the control units 15 and 15e, the weather determination unit 16, the driving determination unit 17, and the notification control unit 18 are implemented by the processing circuit 1601. That is, the automatic driving control device 1 to 1e includes the processing circuit 1601, which derives the automatic driving control amount for controlling the automatic driving of the vehicle 100.
[0257] The processing circuit 1601 may be as follows Fig.16A As shown in the figure is dedicated hardware, you can also Fig. 16B As shown, a CPU (Central Processing Unit) 1605 executes a program stored in a memory 1606 .
[0258] When the processing circuit 1601 is dedicated hardware, the processing circuit 1601 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.
[0259] When the processing circuit 1601 is a CPU 1605, the functions of the information acquisition unit 11, the control amount derivation units 12 to 12e, the monitoring units 14 to 14c, the control units 15 and 15e, the weather determination unit 16, the driving determination unit 17, and the notification control unit 18 are realized by software, firmware, or a combination of software and firmware. That is, the information acquisition unit 11, the control amount derivation units 12 to 12e, the monitoring units 14 to 14c, the control units 15 and 15e, the weather determination unit 16, the driving determination unit 17, and the notification control unit 18 are realized by a processing circuit such as the CPU 1605 or a system LSI (Large-Scale Integration) that executes a program stored in a HDD (Hard Disk Drive) 1602 or a memory 1606. In addition, the program stored in the HDD 1602, the memory 1606, etc. can also be said to be a program that causes the computer to execute the steps or methods of the information acquisition unit 11, the control amount derivation units 12 to 12e, the monitoring units 14 to 14c, the control units 15, 15e, the weather determination unit 16, the driving determination unit 17, and the notification control unit 18. Here, the memory 1606 can be, for example, a nonvolatile or volatile semiconductor memory such as a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic disk, a floppy disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), etc.
[0260] In addition, the functions of the information acquisition unit 11, the control amount derivation units 12 to 12e, the monitoring units 14 to 14c, the control units 15 and 15e, the weather determination unit 16, the driving determination unit 17, and the notification control unit 18 may be partially realized by dedicated hardware, and partially realized by software or firmware. For example, the information acquisition unit 11 may be realized by the processing circuit 1601 as dedicated hardware, and the control amount derivation units 12 to 12e, the monitoring units 14 to 14c, the control units 15 and 15e, the weather determination unit 16, the driving determination unit 17, and the notification control unit 18 may be realized by the processing circuit 1601 reading and executing the program stored in the memory 1606.
[0261] Furthermore, the automatic driving control devices 1 to 1 e have an input interface device 1603 and an output interface device 1604 for performing wired or wireless communication with devices such as sensors, the vehicle control unit 3 or an output device.
[0262] In addition, in the above embodiments 1 to 6, the automatic driving control device 1 to 1e is a vehicle-mounted device mounted on the vehicle 100, and the automatic driving control device 1 to 1e includes an information acquisition unit 11, a control amount derivation unit 12 to 12e, a monitoring unit 14 to 14c, a control unit 15, 15e, a weather determination unit 16, a driving determination unit 17 and a notification control unit 18.
[0263] This is not limited to the above, and part of the information acquisition unit 11, the control amount derivation units 12 to 12e, the monitoring units 14 to 14c, the control units 15, 15e, the weather determination unit 16, the driving determination unit 17, and the notification control unit 18 may be mounted on a vehicle-mounted device of the vehicle 100, and other parts may be provided in a server connected to the vehicle-mounted device via a network, so that the vehicle-mounted device and the server constitute an automatic driving control system.
[0264] Fig.17 It means use Figure 2 The automatic driving control device 1 according to the first embodiment described above includes a diagram showing a configuration example of an automatic driving control system in a server 200 .
[0265] In such Fig.17 In the automatic driving control system shown as an example in FIG, the automatic driving control device 1 and the vehicle-mounted device are connected via the communication device 101 and the communication device 201. The vehicle surrounding information acquired by the sensor is sent to the automatic driving control device 1 on the server 200 via the communication device 101 and the communication device 201. The automatic driving control device 1 derives the automatic driving control amount based on the vehicle surrounding information received from the vehicle-mounted device. Then, the automatic driving control amount derived by the automatic driving control device 1 is sent to the vehicle control unit 3 mounted on the vehicle-mounted device via the communication device 201 and the communication device 101. The vehicle control unit 3 controls the control object device 4 based on the acquired automatic driving control amount.
[0266] Here, as an example, all functions of the automatic driving control device 1 are provided in the server 200, but part of the functions of the automatic driving control device 1 may also be provided in the server 200. For example, the information acquisition unit 11 and the monitoring unit 14 of the automatic driving control device 1 may be provided in the vehicle-mounted device, and other functions of the automatic driving control device 1 may also be provided in the server 200.
[0267] in addition, Fig.17In the embodiment, as an example, the automatic driving control system is a system in which the automatic driving control device 1 involved in the first embodiment is provided in the server 200, but the automatic driving control system may also be a system in which any of the automatic driving control devices 1a to 1e involved in the second to sixth embodiments is provided in the server 200. In the case where the automatic driving control system is set to any of the automatic driving control devices 1a to 1e involved in the second to sixth embodiments is provided in the server 200, Fig.17 In the illustrated configuration example, the server 200 includes a part or all of the functions of the automatic driving control devices 1 a to 1 e .
[0268] Furthermore, the present invention can freely combine the various embodiments within the scope of the invention, or modify any constituent elements of the various embodiments, or omit any constituent elements of the various embodiments.
[0269] Industrial Applicability
[0270] The automatic driving control device involved in the present invention can be applied to an automatic driving control device that performs automatic driving control of a vehicle.
[0271] Description of symbols
[0272] 1~1e Automatic driving control device
[0273] 11 Information Acquisition Department
[0274] 12~12c, 12e Control quantity derivation section
[0275] 121 First control variable derivation unit
[0276] 122 to 122c Second controlled variable derivation unit
[0277] 123 Selection Department
[0278] 13~13c, 13e Machine Learning Model
[0279] 131 First Machine Learning Model
[0280] 132~132c Second machine learning model
[0281] 14~14c Monitoring Department
[0282] 15.15e Control Unit
[0283] 16 Weather Determination Department
[0284] 17 Driving judgment unit
[0285] 18 Notification Control Department
[0286] 21 Camera
[0287] 22 Millimeter Wave Radar
[0288] 23 GNSS
[0289] 24 Vehicle driving sensor
[0290] 3 Vehicle Control Unit
[0291] 4 Control Equipment
[0292] 200 Server
[0293] 101, 201 Communication device
[0294] 1601 Processing Circuit
[0295] 1602 HDD
[0296] 1603 Input interface device
[0297] 1604 Output interface device
[0298] 1605 CPU
[0299] 1606 Memory.
Claims
1. An automatic driving control device, characterized in that: include: an information acquisition unit that acquires a plurality of vehicle periphery information outputted from the plurality of sensors respectively; a control amount derivation unit, which derives an automatic driving control amount based on the plurality of vehicle surrounding information acquired by the information acquisition unit and a machine learning model, and outputs the automatic driving control amount; a monitoring unit configured to determine whether reliability of any of the plurality of pieces of vehicle surrounding information acquired by the information acquisition unit is reduced; as well as a control unit that controls the control amount derivation unit to output the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability when the monitoring unit determines that the reliability of any of the plurality of vehicle surrounding information has decreased, The control unit assigns an information validity flag to all of the plurality of pieces of vehicle periphery information acquired by the information acquisition unit based on a reliability determination result performed by the monitoring unit, The control amount derivation unit derives the automatic driving control amount based on all of the multiple vehicle surrounding information to which the information validity flag is assigned by the control unit and the machine learning model, and outputs the automatic driving control amount.
2. The automatic driving control device according to claim 1, characterized in that: The control amount derivation unit includes: a first control amount derivation unit, which derivates a first automatic driving control amount based on all of the plurality of vehicle surrounding information acquired by the information acquisition unit and a first machine learning model; and a second control amount derivation unit, which derivates a second automatic driving control amount based on a portion of the plurality of vehicle surrounding information acquired by the information acquisition unit and a second machine learning model. When the monitoring unit determines that the reliability of the vehicle surrounding information among the multiple vehicle surrounding information except for a part of the multiple vehicle surrounding information input into the second machine learning model is reduced, the control unit controls the control amount derivation unit so that the control amount derivation unit outputs the second automatic driving control amount.
3. The automatic driving control device according to claim 2, characterized in that: The control amount derivation unit includes a selection unit that selects and outputs either the first automatic driving control amount or the second automatic driving control amount. When the control amount deriving unit is controlled by the control unit to output the second automatic driving control amount, the selection unit selects and outputs the second automatic driving control amount.
4. The automatic driving control device according to claim 2, characterized in that: The second control amount deriving unit derives the second automatic driving control amount when the control amount deriving unit is controlled by the control unit to output the second automatic driving control amount.
5. The automatic driving control device according to claim 1, characterized in that: The plurality of sensors at least include a camera and a millimeter wave radar, As the plurality of vehicle surrounding information, the information acquisition unit acquires at least an image captured by the camera of an object existing around the vehicle and distance information related to the distance to the object measured by the millimeter wave radar, When the difference between the distance to the object obtained based on the captured image acquired by the information acquisition unit and the distance to the object obtained based on the distance information acquired by the information acquisition unit is greater than the radar judgment threshold, the monitoring unit determines that the reliability of the distance information acquired from the millimeter wave radar is reduced.
6. The automatic driving control device according to claim 1, characterized in that: One of the plurality of sensors is a camera, The information acquisition unit acquires, as one of the plurality of vehicle surrounding information, a captured image obtained by the camera capturing the vehicle surroundings. The automatic driving control device includes a weather determination unit, which determines the weather around the vehicle. The monitoring unit determines whether the reliability of the captured image acquired from the camera has decreased based on the weather determined by the weather determination unit.
7. The automatic driving control device according to claim 6, characterized in that: When the weather determination unit determines that there is fog or precipitation around the vehicle, the monitoring unit determines that the reliability of the captured image acquired from the camera is reduced.
8. The automatic driving control device according to claim 1, characterized in that: One of the plurality of sensors is a camera, The information acquisition unit acquires, as one of the plurality of vehicle surrounding information, a captured image obtained by the camera capturing the vehicle surroundings. The monitoring unit determines whether the reliability of the captured image acquired from the camera has decreased based on the brightness of the captured image acquired by the information acquisition unit.
9. The automatic driving control device according to claim 1, characterized in that: One of the plurality of sensors is a camera, The information acquisition unit acquires, as one of the plurality of vehicle surrounding information, a captured image obtained by the camera capturing the vehicle surroundings. The automatic driving control device includes a driving determination unit that determines whether the vehicle is driving. When the travel determination unit determines that the vehicle is traveling and the scenery around the vehicle captured in the captured image acquired by the information acquisition unit does not change, the monitoring unit determines that the reliability of the captured image acquired from the camera is reduced.
10. The automatic driving control device according to claim 2, characterized in that: It includes a notification control unit, which outputs notification information, which shows a message that the reliability of vehicle surrounding information of a part of the multiple vehicle surrounding information is reduced when the control unit controls the control amount derivation unit so that the control amount derivation unit outputs the second automatic driving control amount.
11. The automatic driving control device according to claim 10, characterized in that: The notification information is a message outputting a message indicating that the sensor of the vehicle surrounding information determined to have reduced reliability is unusable.
12. The automatic driving control device according to claim 10, characterized in that: The notification information is a message notifying that a function cannot be used in the automatic driving control due to the presence of the vehicle surrounding information determined to have the reduced reliability.
13. An automatic driving control method, characterized in that: The steps include: The information acquisition unit acquires a plurality of vehicle periphery information outputted from the plurality of sensors respectively; The control amount derivation unit derives the automatic driving control amount based on the plurality of vehicle surrounding information acquired by the information acquisition unit and the machine learning model, and outputs the automatic driving control amount; The monitoring unit determines whether the reliability of any of the plurality of vehicle surrounding information acquired by the information acquisition unit is reduced; as well as The control unit controls the control amount derivation unit, when the monitoring unit determines that the reliability of any of the plurality of vehicle surrounding information is reduced, so that the control amount derivation unit outputs the automatic driving control amount obtained by removing the influence of the vehicle surrounding information determined to have reduced reliability, The control unit assigns an information validity flag to all of the plurality of pieces of vehicle periphery information acquired by the information acquisition unit based on a reliability determination result performed by the monitoring unit, The control amount derivation unit derives the automatic driving control amount based on all of the multiple vehicle surrounding information to which the information validity flag is assigned by the control unit and the machine learning model, and outputs the automatic driving control amount.
