Control device, control method, and computer program product

Abstract

The invention relates to a vehicle control device, and provides a technology capable of effectively monitoring and applying automatic driving control using a machine learning model. The control device includes one or more processors that perform automatic driving control of a vehicle using a machine learning model. The vehicle is configured to be able to be driven by an operator, and the one or more processors are further configured to perform: a process of calculating a reliability related to an inferred result of the machine learning model during execution of the automatic driving control; a process for notifying an operator to urge confirmation of the driving environment of the vehicle when a predetermined first condition indicating that the reliability is reduced is satisfied; and a process of requesting a notification of not releasing the hand to the operator when a predetermined second condition indicating that the reliability is reduced is satisfied in addition to the first condition.

Description

Technical Field

[0001] The present disclosure relates to technology for autonomous driving control of a vehicle using a machine learning model. Background Art

[0002] In recent years, various technologies for utilizing machine learning models as artificial intelligence (AI) have been proposed in the field of vehicle control.

[0003] For example, Patent Document 1 discloses a vehicle control assist system, the purpose of which is to assist in controlling the vehicle using a machine learning model (learned model) that reflects the state of the vehicle, the intention of the driver, etc. In addition, as documents showing the technical level of this technical field, there are the following Patent Documents 2 and 3. Prior art literature Patent Literature

[0004] Patent Document 1: Japanese Patent Application Publication No. 2022-067454 Patent Document 2: Japanese Patent Application Publication No. 2020-004402 Patent Document 3: Japanese Patent Application Publication No. 2021-195933 Summary of the invention Technical problem to be solved by the invention

[0005] In vehicle control, especially in autonomous driving control, it is expected that the control performance can be improved by using machine learning models in various functions of cognition, judgment, and operation. In addition, in autonomous driving control, it is required to properly judge whether control can continue normally. On the other hand, the inference results of machine learning models have probabilistic elements. Therefore, it can be imagined that when using machine learning models for autonomous driving control of vehicles, it is difficult to judge whether control can continue normally.

[0006] In view of the above technical problems, an object of the present disclosure is to provide a technology that can effectively monitor and apply autonomous driving control using a machine learning model. Means for solving technical problems

[0007] A first aspect of the present disclosure relates to a control device for a vehicle.

[0008] The control device involved in the first aspect has one or more processors for performing automatic driving control of a vehicle using a machine learning model. The vehicle is configured to be drivable by an operator, and the one or more processors are further configured to perform the following processing: a process of calculating the reliability related to the inference result of the machine learning model during the execution of the automatic driving control; a process of notifying the operator to urge confirmation of the driving environment of the vehicle when a prescribed first condition indicating a decrease in reliability is satisfied; and a process of notifying the operator to request hands-free operation when a prescribed second condition indicating a decrease in reliability is satisfied in addition to the first condition.

[0009] A second aspect of the present disclosure relates to a control method, which is a control method of a vehicle configured to be drivable by an operator.

[0010] The control method involved in the second aspect executes the following contents through a computer: using a machine learning model to control the automatic driving of the vehicle; in the process of executing the automatic driving control, calculating the reliability related to the inference result of the machine learning model; when a first condition indicating a decrease in reliability is met, notifying the operator to urge him to confirm the driving environment of the vehicle; and when a second condition indicating a decrease in reliability is met in addition to the first condition, notifying the operator to request him to keep his hands on the vehicle.

[0011] A third aspect of the present disclosure relates to a control program that causes a computer to perform control of a vehicle configured to be drivable by an operator.

[0012] The control program involved in the third aspect enables the computer to perform the following processing: processing of autonomous driving control of the vehicle using a machine learning model; processing of calculating the reliability related to the inference result of the machine learning model during the execution of the autonomous driving control; processing of notifying the operator to confirm the driving environment of the vehicle when a prescribed first condition indicating a decrease in reliability is met; and processing of notifying the operator to request him to keep his hands on the vehicle when a prescribed second condition indicating a decrease in reliability is met in addition to the first condition. Effects of the Invention

[0013] According to the present disclosure, the reliability related to the inference result of the machine learning model is calculated during the execution of the automatic driving control. Then, the decrease of the reliability is evaluated in two stages, and when the reliability satisfies the first condition, the operator of the vehicle 1 is notified to urge the confirmation of the driving environment, and when the reliability satisfies the second condition in addition to the first condition, the operator is notified to request the operator to keep the hands-on. In this way, the automatic driving control using the machine learning model can be effectively monitored and applied. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a diagram showing a configuration example related to the automatic driving function of the vehicle involved in this embodiment. Figure 2 It is a diagram showing an example of the configuration of the control device according to the present embodiment. Figure 3 This is a diagram for explaining an example of the reliability calculated in the notification process executed by the processor. Figure 4 This is a flowchart showing an example of processing executed by the processor in the notification processing. Figure 5 It is a diagram showing an example of a control device according to the present embodiment. Figure 6 This is a flowchart showing an example of a process executed by a processor in a notification process according to another embodiment. DETAILED DESCRIPTION

[0015] Hereinafter, embodiments will be described with reference to the drawings.

[0016] 1. Autonomous driving of vehicles The present embodiment relates to a control device for a vehicle. The control device according to the present embodiment performs at least automatic driving control of the vehicle. Figure 1 1 is a block diagram showing a configuration example related to the automatic driving control of the vehicle 1 by the control device according to the present embodiment. The automatic driving control is a concept including not only full automatic driving control but also risk avoidance control, lane keeping assist control, etc.

[0017] The vehicle 1 includes a sensor group 10 , a recognition unit 20 , a planning unit 30 , a control amount calculation unit 40 , and a travel device 50 .

[0018] The sensor group 10 includes an identification sensor 11 for identifying the conditions around the vehicle 1. Examples of the identification sensor 11 include a camera, LIDAR (Laser Imaging Detection and Ranging), and a radar. The sensor group 10 may also include a state sensor 12 for detecting the state of the vehicle 1, a position sensor 13 for detecting the position of the vehicle 1, and the like. Examples of the state sensor 12 include a velocity sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. Examples of the position sensor 13 include a GNSS (Global Navigation Satellite System) sensor.

[0019] The sensor detection information SEN is information obtained by the sensor group 10. For example, the sensor detection information SEN includes an image captured by a camera. As another example, the sensor detection information SEN may include point group information obtained by LIDAR. The sensor detection information SEN may also include vehicle state information indicating the state of the vehicle 1. The sensor detection information SEN may also include position information indicating the position of the vehicle 1.

[0020] The recognition unit 20 receives the sensor detection information SEN. The recognition unit 20 recognizes the surrounding conditions of the vehicle 1 based on the information obtained by the recognition sensor 11. For example, the recognition unit 20 recognizes objects around the vehicle 1. Examples of objects include pedestrians, other vehicles (preceding vehicles, parked vehicles, etc.), white lines, road structures (e.g., guardrails, curbs), fallen objects, traffic lights, intersections, signs, etc. The recognition result information RES shows the recognition result of the recognition unit 20. For example, the recognition result information RES includes object information showing the relative position and relative speed of the object relative to the vehicle 1.

[0021] The planning unit 30 receives the recognition result information RES from the recognition unit 20. In addition, the planning unit 30 may also receive vehicle status information, position information, and pre-generated map information. The map information may be high-precision three-dimensional map information. The planning unit 30 generates a driving plan for the vehicle 1 based on the received information. The driving plan may be used to reach a predetermined destination or to avoid risks. Examples of the driving plan include maintaining the current driving lane, changing lanes, overtaking, turning left or right, steering, accelerating, decelerating, and stopping. In addition, the planning unit 30 generates a target trajectory TRJ required for the vehicle 1 to travel according to the driving plan. The target trajectory TRJ includes a target position and a target speed.

[0022] The control amount calculation unit 40 receives the target trajectory TRJ from the planning unit 30. The control amount calculation unit 40 calculates the control amount CON required to make the vehicle 1 follow the target trajectory TRJ. The control amount CON can also be said to be the control amount required to reduce the deviation between the vehicle 1 and the target trajectory TRJ. The control amount CON includes at least one of a steering control amount, a drive control amount, and a braking control amount. As the steering control amount, a target steering angle, a target torque, a target motor angle, a target motor drive current, etc. can be exemplified. As the drive control amount, a target speed, a target acceleration, etc. can be exemplified. As the braking control amount, a target speed, a target deceleration, etc. can be exemplified.

[0023] The travel device 50 includes a steering device 51, a drive device 52 and a brake device 53. The steering device 51 steers the wheels. For example, the steering device 51 includes an electric power steering (EPS: Electric Power Steering) device. The drive device 52 is a power source that generates a driving force. As the drive device 52, an engine, an electric motor, a hub motor, etc. can be exemplified. The brake device 53 generates a braking force. The travel device 50 receives the control amount CON from the control amount calculation unit 40. The travel device 50 respectively activates the steering device 51, the drive device 52 and the brake device 53 according to the steering control amount, the drive control amount and the brake control amount. As a result, the vehicle 1 travels in a manner of following the target trajectory TRJ.

[0024] The recognition unit 20 includes at least one of a rule-based model and a machine learning model. The rule-based model performs recognition processing based on a predetermined set of rules. As machine learning models, NN (Neural Network), SVM (Support Vector Machine), regression model, decision tree model, etc. can be exemplified. NN can be CNN (Convolutional Neural Network), RNN (Recurrent Neural Network) or a combination thereof. The type, number of layers, and number of nodes of each layer in NN are arbitrary. The machine learning model is pre-generated by machine learning. The recognition unit 20 performs recognition processing by inputting sensor detection information SEN into the model. Recognition result information RES is output from the model, or generated based on the output from the model. Hereinafter, the model included in the recognition unit 20 is also referred to as a "recognition model."

[0025] The planning unit 30 also includes at least one of a rule-based model and a machine learning model. The planning unit 30 performs planning processing by inputting recognition result information RES into the model. The target trajectory TRJ is output from the model or generated based on the output from the model. Hereinafter, the model included in the planning unit 30 is also referred to as a "planning model".

[0026] The control amount calculation unit 40 also includes at least one of a rule-based model and a machine learning model. The control amount calculation unit 40 performs control amount calculation processing by inputting the target trajectory TRJ into the model. The control amount CON is output from the model or generated based on the output from the model. Hereinafter, the model included in the control amount calculation unit 40 is also referred to as a "control amount calculation model".

[0027] Two or more of the recognition unit 20, the planning unit 30, and the control amount calculation unit 40 may also be configured as an integral unit. The recognition unit 20, the planning unit 30, and the control amount calculation unit 40 may all be configured as an integral unit (End-to-End structure). For example, the recognition unit 20 and the planning unit 30 may be configured as an integral unit by a NN that outputs a target trajectory TRJ from the sensor detection information SEN. Even in the case of an integral unit, an intermediate product such as the recognition result information RES and the target trajectory TRJ may be output. For example, in the case where the recognition unit 20 and the planning unit 30 are configured as an integral unit by a NN, the recognition result information RES may be the output of an intermediate layer of the NN.

[0028] The recognition unit 20, the planning unit 30, and the control amount calculation unit 40 constitute an "autonomous driving control unit" that controls the automatic driving of the vehicle 1. In the present embodiment, a machine learning model is used in at least a part of the automatic driving control unit. That is, the automatic driving control unit uses a machine learning model as at least one of the recognition model, the planning model, and the control amount calculation model.

[0029] Figure 2 1 is a block diagram showing an example of the hardware configuration of the control device 101 according to the present embodiment. The control device 101 is mounted on the vehicle 1 and performs at least automatic driving control of the vehicle 1. That is, the control device 101 has at least the function of the automatic driving control unit described above.

[0030] The control device 101 is configured to be able to communicate with the sensor group 10 , the travel device 50 , the display unit 60 , the speaker 70 , and the actuator group.

[0031] The control device 101 is a computer including one or more processors 110 (hereinafter, simply referred to as processors 110 ) and one or more storage devices 120 (hereinafter, simply referred to as storage devices 120 ).

[0032] The processor 110 performs various processing. As the processor 110, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc. can be exemplified. The recognition unit 20, the planning unit 30, and the control amount calculation unit 40 can be implemented by a single processor 110 or by different processors 110. The storage device 120 stores various information. As the storage device 120, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a volatile memory, a non-volatile memory, etc. can be exemplified.

[0033] The storage device 120 stores a computer program 130 and model data 140 .

[0034] The computer program 130 is executed by the processor 110. The processor 110 executing the computer program 130 may cooperate with the storage device 120 to realize various processes by the control device 101. The computer program 130 may also be recorded on a computer-readable recording medium.

[0035] The model data 140 is data of the recognition model, the planning model, and the control amount calculation model. In the execution of the automatic driving control, the processor 110 configures the recognition unit 20, the planning unit 30, and the control amount calculation unit 40 by using the model read from the model data 140. As described above, in the present embodiment, the machine learning model is used as at least one of the recognition model, the planning model, and the control amount calculation model.

[0036] The display unit 60 performs various displays. The display unit 60 is configured to be controllable by the control device 101. Examples of the display unit 60 include an instrument panel display, a multi-information display, a head-up display, an indicator, and the like.

[0037] The speaker 70 emits various sounds. The speaker 70 is configured to be controllable by the control device 101 .

[0038] In addition, in the present embodiment, the vehicle 1 is configured to be driven by an operator. For example, the vehicle 1 is provided with a driving operation device (steering wheel, accelerator pedal, brake pedal, etc.) that accepts driving operation by the operator. In this case, the operator is typically a driver riding in the vehicle 1. However, the operator may also be a remote operator who remotely operates the vehicle 1. The vehicle 1 may also be configured so that, when performing automatic driving control, the operator can drive the vehicle in an override manner by operating the driving operation device.

[0039] 2. Notification Processing As described above, the control device 101 involved in this embodiment uses a machine learning model to perform automatic driving control of the vehicle 1. In automatic driving control, it is required to appropriately judge whether the control can be continued normally. On the other hand, the inference result of the machine learning model is calculated based on probabilistic inference based on training data. Therefore, there is a possibility that errors will occur in the inference result, and it is considered inappropriate to judge whether the control can be continued normally only based on the inference result.

[0040] Therefore, in the control device 101 involved in this embodiment, the processor 110 calculates the reliability related to the inference result of the machine learning model during the execution of the automatic driving control. In addition, the processor 110 is configured to perform a process of notifying the operator of the vehicle 1 based on the calculated reliability (hereinafter referred to as "notification process"). The notification process is described in detail below.

[0041] 2-1. Reliability The reliability calculated in the notification process can be the confidence of the machine learning model for the inference result. Figure 3 , illustrating an example of the reliability calculated in the notification processing.

[0042] Figure 3 (A) shows the case where a machine learning model is used as the control amount calculation model 144. Figure 3 In (A), the control amount calculation model 144 is configured to take the target trajectory TRJ as input and output three control amounts CON1, CON2 and CON3 as inference results. For example, the three control amounts CON1, CON2 and CON3 are respectively a steering control amount, a driving control amount and a braking control amount. The control amount calculation unit 40 outputs the control amount CON including the three control amounts CON1, CON2 and CON3.

[0043] exist Figure 3In (A), the control amount calculation model 144 is further configured to output the confidence level for each of the three control amounts CON1, CON2, and CON3. At this time, the processor 110 can calculate the reliability related to the inference result of the control amount calculation model 144 based on each confidence level. For example, the processor 110 calculates the minimum confidence level as the reliability. In this case, Figure 3 In (A), the processor 110 calculates the reliability of the inference result of the control amount calculation model 144 to be 30%. In addition, for example, the processor 110 calculates the average value of each confidence as the reliability. In this case, Figure 3 In (A), the processor 110 calculates that the reliability associated with the inference result of the control amount calculation model 144 is 70%.

[0044] When the automatic driving control unit uses a plurality of machine learning models, the processor 110 may calculate the reliability for all of the plurality of machine learning models in the same manner as described above. Alternatively, the processor 110 may calculate the reliability for each machine learning model as described below.

[0045] Figure 3 (B) shows the case where a machine learning model is used as the recognition model 142 and the planning model 143. Figure 3 In (B), the recognition model 142 is configured to take the sensor detection information SEN as input and output two recognition result information RES1 and RES2 as inference results. For example, the two recognition result information RES1 and RES2 are the recognition results of other vehicles and the recognition results of white lines, respectively. The recognition unit 20 outputs the recognition result information RES including the two recognition result information RES1 and RES2. The planning model 143 is configured to take the recognition result information RES as input and output the target trajectory TRJ as the inference result. The planning unit 30 outputs the target trajectory TRJ output from the planning model 143.

[0046] exist Figure 3 In (B), the recognition model 142 is further configured to output the confidence for each of the two recognition result information RES1 and RES2. In addition, the planning model 143 is further configured to output the confidence for the target trajectory TRJ. At this time, the processor 110 can calculate the reliability associated with the inference result of the recognition model 142 and the reliability associated with the inference result of the planning model 143 according to each confidence. For example, the processor 110 calculates that the reliability associated with the inference result of the recognition model 142 is 70%, and calculates that the reliability associated with the inference result of the planning model 143 is 90%.

[0047] As described above, the processor 110 can calculate the reliability. As another example of reliability, when a machine learning model outputs multiple inference results, the reliability can be calculated by using the degree of deviation between the multiple inference results as an indicator. For example, when a machine learning model is used as a planning model, the reliability can be calculated by using the deviation in the time series between the output multiple target trajectories TRJ as an indicator.

[0048] 2-2. Processing Figure 4 1 is a flowchart showing an example of processing performed by the processor 110 in the notification processing. During the startup of the control device 101, Figure 4 The processing related to the flowchart shown is repeatedly executed in a predetermined processing cycle.

[0049] First, in step S100, the processor 110 calculates the reliability of the inference result of the machine learning model used by the automatic driving control unit. For example, as described above, the processor 110 obtains the confidence for each inference result and calculates the reliability based on the obtained confidence.

[0050] Next, in step S110, the processor 110 determines whether the calculated reliability has dropped below a predetermined first threshold. When the reliability is calculated for each machine learning model, the processor 110 may also be configured to determine whether any reliability has dropped below the first threshold.

[0051] If the reliability is greater than the first threshold (step S110: No), the processor 110 ends the current process without notifying the operator. If the reliability decreases below the first threshold (step S110: Yes), the process proceeds to step S120.

[0052] In step S120, the processor 110 determines whether the calculated reliability has dropped below a predetermined second threshold value that is smaller than the first threshold value. When the reliability is calculated for each machine learning model, the processor 110 may also be configured to determine whether any reliability among the reliabilities determined to be less than the first threshold value has dropped below the second threshold value.

[0053] When the reliability is above the second threshold value (step S120: "No"), that is, when the reliability decreases to less than the first threshold value but does not decrease to less than the second threshold value, the processor 110 notifies the operator to urge confirmation of the driving environment of the vehicle 1 (step S130). The so-called "confirmation of the driving environment" refers to the confirmation of surrounding objects, the confirmation of surrounding traffic conditions, the confirmation of the driving state of the vehicle, the confirmation of the status of automatic driving control (driving plan or target trajectory TRJ), etc. For example, the processor 110 causes the speaker 70 to output a sound such as "Please confirm the surroundings of the vehicle and pay attention to driving based on automatic driving." In addition, for example, the processor 110 lights up an indicator indicating that it is in a state of requesting confirmation of the driving environment. In addition, for example, the processor 110 displays the driving plan or target trajectory TRJ of the automatic driving control on the instrument panel display or the multi-information display.

[0054] After step S130, processor 110 ends the current processing.

[0055] By issuing a notification urging confirmation of the driving environment, the operator of the vehicle 1 can be expected to confirm the driving environment and monitor whether the automatic driving control is being performed normally. The operator of the vehicle 1 can prepare for a situation in which the automatic driving control is not being performed normally.

[0056] When the reliability is less than the second threshold (step S120: "Yes"), that is, when the reliability decreases to less than the first threshold and then further decreases to less than the second threshold, the processor 110 notifies the operator of a request to keep the hands on the vehicle (step S140). The "hands on" mentioned here refers to the state of the operator when driving the vehicle 1, and various states can be envisioned according to the environment in which the present embodiment is applied. Typically, it is a state of sitting in the driver's seat and holding the steering wheel. For example, the processor 110 causes the speaker 70 to output a sound such as "Please be prepared to be able to perform driving operations immediately, and pay attention to driving based on automatic driving." In addition, for example, the processor 110 lights up an indicator indicating that it is in a state where it is requested to keep the hands on the vehicle. In addition, for example, the processor 110 causes the instrument panel display or the multi-information display to display a window warning that it is in a state where it is requested to keep the hands on the vehicle.

[0057] After step S140, processor 110 ends the current processing.

[0058] When the reliability is reduced to a considerable extent, it can be considered that there is a high possibility that the automatic driving control suddenly cannot be performed normally. By making a notification requesting hands-free operation, it can be expected that the operator of the vehicle 1 will terminate the automatic driving control or override the automatic driving control and immediately start driving the vehicle 1 when it is determined that the automatic driving control is not performed normally.

[0059] As described above, the notification process is executed in the control device 101 according to the present embodiment. In addition, the control method according to the present embodiment is implemented by executing the process by the processor 110 in this way. In addition, the control program according to the present embodiment is realized by the computer program 130 that causes the processor 110 to execute the process in this way.

[0060] In addition, it can be considered that the more complex the traffic conditions around the vehicle 1, the more significant the change in reliability in the execution of the automatic driving control. For example, in a situation where there are a large number of objects around the vehicle 1, it can be predicted that the reliability is likely to change due to frequent object recognition. In such a situation, the possibility of a sharp change in reliability is high, so it is preferable to notify the operator in advance.

[0061] Therefore, the processor 110 may be configured to change the first threshold value or the second threshold value according to the complexity of the traffic conditions around the vehicle 1 during the notification process.

[0062] The complexity of the traffic situation can be indexed by the type of the road being traveled, the traffic volume, the geographical conditions, etc. For example, it is assumed that the complexity of the traffic situation is low when the vehicle 1 is traveling on a highway, and the complexity of the traffic situation is high when the vehicle 1 is traveling on a street. In addition, for example, it is assumed that the more traffic volume around the vehicle 1, the higher the complexity of the traffic situation. In addition, for example, it is assumed that the complexity of the traffic situation is low when the weather around the vehicle 1 is sunny, and the complexity of the traffic situation is high when the weather around the vehicle 1 is rainy.

[0063] The processor 110 may obtain the surrounding environment information of the vehicle 1 as an index, and calculate the complexity of the traffic conditions around the vehicle 1 based on the obtained surrounding environment information. The processor 110 obtains the surrounding environment information from, for example, the sensor detection information SEN or the recognition result information RES. The processor 110 may evaluate the complexity of the traffic conditions in stages, or may evaluate the complexity of the traffic conditions using a continuous numerical value.

[0064] When the complexity of the traffic situation is high, the processor 110 changes the first threshold or the second threshold to a larger value. Figure 5 FIG. 1 is a diagram showing an embodiment in which the first threshold or the second threshold is changed according to the complexity of the traffic condition. Figure 5 In FIG. 1 , an example of the reliability changing over time and the first and second thresholds corresponding to the complexity of the traffic conditions are shown. Figure 5 In the embodiment, the processor 110 changes the first threshold and the second threshold so as to increase them respectively when the complexity of the traffic condition is high.

[0065] like Figure 5As shown, by changing the first threshold or the second threshold according to the complexity of the traffic situation, the time of notifying the operator to confirm the driving environment of the vehicle 1 (confirmation notification) or requesting the operator to keep the hands on the vehicle 1 (hands on notification) can be advanced. In this way, the operator can be notified in advance in a situation where the reliability is likely to change.

[0066] In addition, the processor 110 may be configured to change the first threshold value or the second threshold value according to the state of the vehicle 1. For example, the processor 110 may change the first threshold value or the second threshold value to a larger value when the speed of the vehicle 1 is high.

[0067] 2-3. Other embodiments As described above, in the notification process, the decrease in reliability is evaluated in two stages, and when the condition representing the first stage (first condition) is met, a notification is made to urge confirmation of the driving environment of the vehicle 1, and when the condition representing the second stage (second condition) is met, a notification is made to request not to let go. In the above description, the first condition is that the reliability is less than a first threshold, and the second condition is that the reliability is further less than a second threshold that is smaller than the first threshold.

[0068] In addition, the automatic driving control unit uses the machine learning model as at least one of the recognition model, the planning model, and the control amount calculation model. In particular, in the automatic driving control unit, the use of the machine learning model as the upstream recognition model and the planning model has a great impact. In addition, the processor 110 can calculate the reliability of the recognition model and the planning model respectively.

[0069] Therefore, when the machine learning model is used as the recognition model and the planning model respectively, the reliability related to the inference result of the recognition model (hereinafter referred to as the "first reliability") and the reliability related to the inference result of the planning model (hereinafter referred to as the "second reliability") can be considered to evaluate the reduction in reliability in two stages. From this point of view, the notification process can also be implemented as follows.

[0070] Figure 6 1 is a flowchart showing an example of a process executed by the processor 110 in the notification process according to another embodiment. During the startup of the control device 101, Figure 6 The processing related to the flowchart shown is repeatedly executed in a predetermined processing cycle.

[0071] First, in step S200 , the processor 110 calculates a first reliability and a second reliability.

[0072] Next, in step S210 , the processor 110 determines whether any one of the first reliability and the second reliability has dropped below a predetermined threshold.

[0073] When both the first reliability and the second reliability are above the threshold value (step S210: No), the processor 110 ends the current processing without notifying the operator. When either the first reliability or the second reliability decreases below the predetermined threshold value (step S210: Yes), the processing proceeds to step S220.

[0074] In step S220 , the processor 110 determines whether both the first reliability and the second reliability are reduced to less than a prescribed threshold.

[0075] When either the first reliability or the second reliability is above the threshold value (step S220: No), that is, when only one of the first reliability and the second reliability drops below the threshold value, the processor 110 notifies the operator to confirm the driving environment of the vehicle 1 (step S230).

[0076] After step S230, the processor 110 ends the process.

[0077] When both the first reliability and the second reliability are lowered to be lower than the threshold value (step S220: Yes), the processor 110 notifies the operator to request the operator to keep the hands-on operation (step S240).

[0078] After step S240, the processor 110 ends the process.

[0079] As described above, the processor 110 may also perform the notification process according to another embodiment. In the notification process according to another embodiment, the first condition is that either the first reliability or the second reliability is less than a predetermined threshold, and the second condition is that both the first reliability and the second reliability are less than the threshold.

[0080] 3. Effect As described above, according to the present embodiment, the reliability related to the inference result of the machine learning model is calculated during the execution of the automatic driving control. Then, the reduction in reliability is evaluated in two stages, and when the reliability satisfies the first condition, a notification urging the operator of the vehicle 1 to confirm the driving environment is issued, and when the reliability satisfies the second condition in addition to the first condition, a notification requesting the operator to keep the hands on the vehicle is issued. By issuing a notification urging the confirmation of the driving environment, the operator can confirm the driving environment and monitor whether the automatic driving control is proceeding normally. In addition, by issuing a notification requesting the hands on, it is possible to prepare for a situation where it is determined that the automatic driving control is not proceeding normally, and the operator can be put in a state where he can start driving the vehicle 1 immediately. In this way, according to the present embodiment, the automatic driving control using the machine learning model can be effectively monitored and applied. [Explanation of Reference Numerals]

[0081] 1 vehicle, 10 sensor group, 20 recognition unit, 30 planning unit, 40 control amount calculation unit, 50 travel device, 60 display unit, 70 speaker, 101 control device, 110 processor, 120 storage device, 130 computer program, 142 recognition model, 143 planning model, 144 control amount calculation model.

Claims

1. A control device, which is a control device of a vehicle, having one or more processors for performing autonomous driving control of the vehicle using a machine learning model, The vehicle is configured to be driven by an operator, The one or more processors are further configured to perform the following processing: In the process of executing the autonomous driving control, a process of calculating the reliability related to the inference result of the machine learning model; When a predetermined first condition indicating a decrease in the reliability is satisfied, performing a process of notifying the operator to urge him to check the driving environment of the vehicle; and When a predetermined second condition indicating a decrease in the reliability is satisfied in addition to the first condition, a process of notifying the operator to request the operator to keep his hands on the phone is performed.

2. The control device according to claim 1, characterized in that: The one or more processors calculate the reliability based on the confidence of the machine learning model for the inference result.

3. The control device according to claim 1 or 2, characterized in that: The first condition is that the reliability is less than a specified first threshold, The second condition is that the reliability is less than a predetermined second threshold value that is smaller than the first threshold value.

4. The control device according to claim 3, characterized in that: The one or more processors are further configured to perform: Acquiring information about the surrounding environment of the vehicle; A process of calculating the complexity of the traffic conditions around the vehicle based on the surrounding environment information; and The process of changing the first threshold or the second threshold according to the complexity.

5. The control device according to claim 1 or 2, characterized in that: The machine learning model includes a recognition model for recognizing the surrounding conditions of the vehicle and a planning model for generating a target trajectory of the autonomous driving control. The reliability includes a first reliability related to the inference result of the recognition model and a second reliability related to the inference result of the planning model, The first condition is that either the first reliability or the second reliability is less than a predetermined threshold value. The second condition is that both the first reliability and the second reliability are smaller than the threshold value.

6. A control method, which is a control method for a vehicle, the vehicle being configured to be drivable by an operator, Execute the following from your computer: utilizing a machine learning model to perform autonomous driving control of the vehicle; In the process of performing the autonomous driving control, calculating the reliability associated with the inference result of the machine learning model; When a predetermined first condition indicating a decrease in the reliability is satisfied, notifying the operator to confirm the driving environment of the vehicle; and When a predetermined second condition indicating a decrease in the reliability is satisfied in addition to the first condition, a notification requesting the operator to keep his hands on the device is made.

7. A computer program product comprising a control program for causing a computer to execute control of a vehicle, the vehicle being configured to be drivable by an operator, the control program being configured to cause the computer to execute the following processing: Processing of autonomous driving control of the vehicle using a machine learning model; In the process of executing the autonomous driving control, a process of calculating the reliability related to the inference result of the machine learning model; When a predetermined first condition indicating a decrease in the reliability is satisfied, performing a process of notifying the operator to urge him to check the driving environment of the vehicle; and When a predetermined second condition indicating a decrease in the reliability is satisfied in addition to the first condition, a process of notifying the operator to request the operator to keep his hands on the phone is performed.

Images