Control system

Abstract

A control system mounted on a vehicle is provided. The control system is provided with one or more processors that execute one or more control functions in accordance with a driving environment or a request of a vehicle. The one or more processors are further configured to execute: a first process of determining whether or not a machine learning model is used in a control function being executed; and a second process for notifying the operator of the vehicle that the control function using the machine learning model is being executed when it is determined that the machine learning model is being used. As a result, it is possible to reduce erroneous recognition of an operator of the vehicle with respect to whether or not a control function using a machine learning model is being used.

Description

Technical Field

[0001] The present disclosure relates to a control system mounted on a vehicle. Background Art

[0002] In recent years, the application of machine learning models as artificial intelligence (AI) has been promoted in various fields. In addition, various fields are considering technologies for effectively applying AI.

[0003] Patent document 1 discloses an information processing device having a user information acquisition unit for determining a usage scenario in which a user utilizes AI, and an AI selection unit for selecting AI corresponding to the determined usage scenario. In addition, the following patent document 2 is a document representing the technical level in this technical field.

[0004] Prior art literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-056970

[0006] Patent Document 2: Japanese Patent Application Publication No. 2019-012255 Summary of the invention

[0007] Problems to be solved by the invention

[0008] It is expected that the application of machine learning models will also be promoted in the field of vehicle control in the future. The vehicle operator may consider using the vehicle control function that uses the machine learning model with certain consent. On the other hand, a variety of functions are implemented in vehicle control, and it is conceivable that the utilization of the machine learning model in each control function will be different. Therefore, the vehicle operator may misidentify (misrecognize) whether the control function that uses the machine learning model is being used. Such misrecognition may cause the vehicle operator to encounter unforeseen situations.

[0009] In view of the above problems, an object of the present disclosure is to reduce the misunderstanding of a vehicle operator as to whether a control function utilizing a machine learning model is being used.

[0010] Technical solutions to solve problems

[0011] One aspect of the present disclosure relates to a control system mounted on a vehicle.

[0012] The control system includes one or more processors for executing one or more control functions according to a driving environment or request of a vehicle. The one or more processors are further configured to execute: a first process of determining whether a machine learning model is being used in the control function being executed; and a second process of notifying an operator of the vehicle that a control function using the machine learning model is being executed when it is determined that the machine learning model is being used.

[0013] Effects of the Invention

[0014] According to the present disclosure, it is determined whether a machine learning model is being used in a control function being executed. Furthermore, when it is determined that a machine learning model is being used, the operator of the vehicle is notified that a control function using the machine learning model is being executed. This can reduce the misunderstanding of the operator of the vehicle as to whether a control function using a machine learning model is being used. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a diagram showing a configuration example of a vehicle related to an automatic driving function according to the present embodiment.

[0016] Figure 2 This is a diagram showing an example of the configuration of the automatic driving system involved in this embodiment.

[0017] Figure 3 This is a diagram showing an example of the configuration of an automatic driving control unit that is determined to utilize a machine learning model.

[0018] Figure 4 This is a diagram for explaining an example of notification according to the present embodiment.

[0019] Figure 5 This is a diagram showing an example in which notification is changed according to the degree of contribution.

[0020] Figure 6 This is a flowchart showing an example of a process executed by a processor.

[0021] Description of Reference Numerals

[0022] 1 vehicle; 10 sensor group; 20 identification unit; 30 planning unit; 40 control amount calculation unit; 50 driving device; 60 display unit; 70 speaker; 80 actuator group; 100 automatic driving system; 110 processor; 120 storage device; 130 computer program; 140 model data. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments will be described with reference to the drawings.

[0024] 1. Autonomous Driving System

[0025] The present embodiment relates to an automatic driving system for a vehicle. The automatic driving system according to the present embodiment is mounted on a vehicle and performs an automatic driving function according to the driving environment or request of the vehicle. Figure 1 This is a block diagram showing an example of the configuration of a vehicle 1 associated with an automatic driving function according to the automatic driving system involved in this embodiment. Automatic driving means that at least one of steering, acceleration and deceleration of the vehicle 1 is automatically performed without relying on the driving operation of the operator. The automatic driving function is a concept that includes not only full automatic driving control but also risk avoidance control and lane keeping assist control. The operator can be either a driver riding in the vehicle 1 or a remote operator who remotely operates the vehicle 1.

[0026] The vehicle 1 includes a sensor group 10 , a recognition unit 20 , a planning unit 30 , a control amount calculation unit 40 , and a travel device 50 .

[0027] The sensor group 10 includes an identification sensor 11 used to identify the conditions around the vehicle 1. Examples of the identification sensor 11 include a camera, LIDAR (Laser Imaging Detection and Ranging), and a radar. The sensor group 10 may also include a state sensor 12 for detecting the state of the vehicle 1, and a position sensor 13 for detecting the position of the vehicle 1. Examples of the state sensor 12 include a velocity sensor, an acceleration sensor, a yaw rate sensor, and a steering angle sensor. Examples of the position sensor 13 include a GNSS (Global Navigation Satellite System) sensor.

[0028] The sensor detection information SEN is information obtained by the sensor group 10. For example, the sensor detection information SEN includes an image captured by a camera. As another example, the sensor detection information SEN may also include point cloud information obtained by LIDAR. The sensor detection information SEN may also include vehicle state information indicating the state of the vehicle 1. The sensor detection information SEN may also include position information indicating the position of the vehicle 1.

[0029] The recognition unit 20 receives the sensor detection information SEN. The recognition unit 20 recognizes the driving environment of the vehicle 1 based on the information obtained by the recognition sensor 11. For example, the recognition unit 20 recognizes objects around the vehicle 1. Examples of objects include pedestrians, other vehicles (the vehicle in front, parked vehicles, etc.), white lines, road structures (such as guardrails, curbs), fallen objects, signal lights, intersections, road signs, etc. In addition, for example, the recognition unit 20 estimates the own position of the vehicle 1. The recognition result information RES represents the recognition result obtained by the recognition unit 20. For example, the recognition result information RES includes object information indicating the relative position and relative speed of the object relative to the vehicle 1.

[0030] The planning unit 30 receives the recognition result information RES from the recognition unit 20. In addition, the planning unit 30 may also receive vehicle status information, position information, and pre-generated map information. The map information may be high-precision three-dimensional map information. The planning unit 30 generates a driving plan for the vehicle 1 based on the received information. The driving plan may be a plan for reaching a predetermined destination or a plan for avoiding risks. Examples of driving plans include maintaining the current driving lane, changing lanes, overtaking, right and left turns, steering, accelerating, decelerating, and stopping. Furthermore, the planning unit 30 generates a target trajectory TRJ required for the vehicle 1 to travel according to the driving plan. The target trajectory TRJ includes a target position and a target speed. The driving plan and target trajectory TRJ generated by the planning unit 30 can be said to be control judgments in the automatic driving function.

[0031] The control amount calculation unit 40 receives the target trajectory TRJ from the planning unit 30. The control amount calculation unit 40 calculates the control amount CON required for the vehicle 1 to follow the target trajectory TRJ. The control amount CON can also be said to be the control amount required to reduce the deviation between the vehicle 1 and the target trajectory TRJ. The control amount CON includes at least one of a steering control amount, a drive control amount, and a braking control amount. As a steering control amount, a target steering angle, a target torque, a target motor angle, a target motor drive current, etc. can be exemplified. As a drive control amount, a target speed, a target acceleration, etc. can be exemplified. As a braking control amount, a target speed, a target deceleration, etc. can be exemplified.

[0032] The travel device 50 includes a steering device 51, a drive device 52, and a brake device 53. The steering device 51 steers the wheels. For example, the steering device 51 includes an electric power steering (EPS: Electric Power Steering) device. The drive device 52 is a power source that generates a driving force. As the drive device 52, an engine, an electric motor, an in-wheel motor, etc. can be exemplified. The brake device 53 generates a braking force. The travel device 50 receives the control amount CON from the control amount calculation unit 40. The travel device 50 operates the steering device 51, the drive device 52, and the brake device 53 according to each of the steering control amount, the drive control amount, and the brake control amount. As a result, the vehicle 1 travels in a manner of following the target trajectory TRJ.

[0033] The recognition unit 20 includes at least one of a rule-based model and a machine learning model. The rule-based model performs recognition processing based on a predetermined rule group. As machine learning models, NN (Neural Network), SVM (Support Vector Machine), regression model, decision tree model, etc. can be exemplified. NN can be CNN (Convolutional Neural Network), RNN (Recurrent Neural Network) or a combination thereof. The type of each layer in NN, the number of layers, and the number of nodes are arbitrary. The machine learning model is pre-generated by machine learning. The recognition unit 20 performs recognition processing by inputting sensor detection information SEN into the model. The recognition result information RES is output from the model or generated based on the output from the model. The recognition unit 20 can also be configured to perform multiple recognition processes using multiple models. For example, the recognition unit 20 can be configured to include a rule-based model for estimating its own position and a machine learning model for object recognition.

[0034] The planning unit 30 also includes at least one of a rule-based model and a machine learning model. The planning unit 30 performs planning processing by inputting recognition result information RES into the model. The target trajectory TRJ is output from the model or is generated based on the output from the model. The planning unit 30 may also be configured to perform planning processing using multiple models. For example, the planning unit 30 may also be configured to include a machine learning model for generating a driving plan, and a rule-based model for generating a target trajectory TRJ. Furthermore, the planning unit 30 may also be configured to be able to switch models according to the driving environment, request, etc. of the vehicle 1. For example, the planning unit 30 may also be configured to use a rule-based model when the vehicle 1 is traveling on a highway in the generation of the driving plan, and on the other hand, use a machine learning model when the vehicle 1 is traveling on a general road.

[0035] The control amount calculation unit 40 also includes at least one of a rule-based model and a machine learning model. The control amount calculation unit 40 performs control amount calculation processing by inputting the target trajectory TRJ into the model. The control amount CON is output from the model or generated based on the output from the model. The control amount calculation unit 40 can also be configured to perform multiple control amount calculation processes using multiple models. For example, the control amount calculation unit 40 can also be configured to include a machine learning model for calculating the steering control amount, and a rule-based model for calculating the drive control amount and the braking control amount. Furthermore, the control amount calculation unit 40 can also be configured to be able to switch models according to the driving environment, request, etc. of the vehicle 1.

[0036] Two or more of the recognition unit 20, the planning unit 30 and the control amount calculation unit 40 may be integrally formed. The recognition unit 20, the planning unit 30 and the control amount calculation unit 40 may also be all integrally formed (end-to-end structure). For example, the recognition unit 20 and the planning unit 30 may also be integrally formed by a NN that outputs a target trajectory TRJ based on the sensor detection information SEN. Even in the case of an integrated structure, an intermediate product such as the recognition result information RES and / or the target trajectory TRJ may be output. For example, in the case where the recognition unit 20 and the planning unit 30 are integrally formed by a NN, the recognition result information RES may be the output of an intermediate layer of the NN.

[0037] The recognition unit 20 , the planning unit 30 , and the control amount calculation unit 40 constitute an “automatic driving control unit” that executes the automatic driving function of the vehicle 1 .

[0038] Figure 2 1 is a block diagram showing an example of the hardware configuration of the automatic driving system 100 according to the present embodiment. The automatic driving system 100 has at least the function of the automatic driving control unit described above.

[0039] The automatic driving system 100 includes a processing unit 101, a sensor group 10, a travel device 50, a display unit 60, a speaker 70, and an actuator group 80. The processing unit 101 is configured to be able to communicate with the sensor group 10, the travel device 50, the display unit 60, the speaker 70, and the actuator group 80.

[0040] The processing unit 101 is a computer including one or more processors 110 (hereinafter, simply referred to as processors 110 ) and one or more storage devices 120 (hereinafter, simply referred to as storage devices 120 ).

[0041] The processor 110 performs various processing. As the processor 110, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc. can be exemplified. The recognition unit 20, the planning unit 30, and the control amount calculation unit 40 can be implemented by a single processor 110 or by separate processors 110. The storage device 120 stores various information. As the storage device 120, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a volatile memory, a non-volatile memory, etc. can be exemplified.

[0042] The storage device 120 stores a computer program 130 and model data 140 .

[0043] The computer program 130 is executed by the processor 110. Various processes in the automatic driving system 100 may be implemented through cooperation between the processor 110 and the storage device 120 that executes the computer program 130. The computer program 130 may also be recorded in a computer-readable recording medium.

[0044] The model data 140 is data of the model included in the recognition unit 20, the planning unit 30, and the control amount calculation unit 40. The model data 140 is stored in the storage device 120. In the execution of the automatic driving function, the processor 110 selects and utilizes a model from the model data 140 to constitute the recognition unit 20, the planning unit 30, and the control amount calculation unit 40. In particular, the processor 110 may also be configured to determine the model to be selected based on the driving environment of the vehicle 1, the request, etc. In this way, the switching of the models included in the recognition unit 20, the planning unit 30, and the control amount calculation unit 40 is achieved.

[0045] The display unit 60 is mounted on the vehicle 1 and performs various displays. The display unit 60 is configured to be controllable by the processing unit 101. Examples of the display unit 60 include an instrument panel display, a multi-information display, a head-up display, and indicators.

[0046] The speaker 70 is mounted on the vehicle 1 and emits various sounds. The speaker 70 is configured to be controllable by the processing unit 101 .

[0047] The actuator group 80 includes actuators of various devices provided in the vehicle 1 . For example, the actuator group 80 includes a vibration actuator for vibrating the steering wheel and / or a reaction torque actuator for generating a reaction torque on the steering wheel. The actuator group 80 is configured to be controllable by the processing unit 101 .

[0048] 2. Notice to the vehicle operator

[0049] In the autonomous driving system 100, the autonomous driving function executed by the processor 110 sometimes utilizes a machine learning model and sometimes utilizes only a rule-based model. In addition, it is assumed that the utilization of the machine learning model in each of the recognition unit 20, the planning unit 30, and the control amount calculation unit 40 will vary depending on the driving environment of the vehicle 1, the request, etc.

[0050] In the automatic driving system 100 according to the present embodiment, the processor 110 determines whether the machine learning model is being used (or not used) in the automatic driving function being executed. Furthermore, when it is determined that the machine learning model is being used, the processor 110 notifies the operator of the vehicle 1 that the automatic driving function using the machine learning model is being executed. These processes are described below.

[0051] 2-1. Determine whether machine learning models are being used

[0052] The processor 110 can determine whether the machine learning model is being used from the following viewpoints based on the respective configurations of the recognition unit 20 , the planning unit 30 , and the controlled variable calculation unit 40 .

[0053] The first viewpoint is whether the output of the machine learning model is used as a control judgment or control amount. That is to say, according to this viewpoint, the processor 110 determines whether the machine learning model is being used based on the utilization status of the machine learning model in the planning unit 30 or the control amount calculation unit 40. For example, when the output of the machine learning model is used as a driving plan or a target trajectory TRJ in the planning unit 30, the processor 110 determines that the machine learning model is being used. In addition, for example, when the output of the machine learning model is used as a steering control amount, a drive control amount, or a braking control amount in the control amount calculation unit 40, the processor 110 determines that the machine learning model is being used.

[0054] Figure 3 (A) and (B) of FIG. 1 each represent an example of a configuration of an automatic driving control unit that is determined to utilize a machine learning model based on the first viewpoint. Figure 3 In (A), the planning unit 30 is composed of a machine learning model 31 and a rule-based model 32. In addition, the output of the machine learning model 31 becomes the driving plan PLN. Figure 3 In (B), the control amount calculation unit 40 is composed of a machine learning model 41, a machine learning model 42, and a rule-based model 43. In addition, the outputs of the machine learning model 41 and the machine learning model 42 become part of the control amount CON. For example, the machine learning model 41 is a model for calculating the steering control amount, and the machine learning model 42 is a model for calculating the driving control amount.

[0055] The second point of view is whether the information of the driving environment of the vehicle 1 identified by the machine learning model is used to make a control judgment or calculate the control amount. That is, according to this point of view, the processor 110 determines whether the machine learning model is being used based on the utilization of the recognition result information RES output based on the machine learning model. For example, when the planning unit 30 uses the object information based on the output of the machine learning model to generate the driving plan PLN or the target trajectory TRJ, the processor 110 determines that the machine learning model is being used.

[0056] Figure 3 (C) shows an example of the configuration of an automatic driving control unit determined to utilize a machine learning model based on the second viewpoint. Figure 3 In (C), the recognition unit 20 is composed of a machine learning model 21, a rule-based model 22, and a rule-based model 23. In addition, in the planning unit 30, the rule-based model 33 uses the recognition result information RES based on the output of the machine learning model 21 to generate a driving plan PLN.

[0057] As described above, based on the above viewpoints, the processor 110 can determine whether the machine learning model is being used in the autonomous driving function being executed based on the respective configurations of the recognition unit 20, the planning unit 30, and the control amount calculation unit 40. The processor 110 can also combine the above viewpoints to determine whether the machine learning model is being used.

[0058] 2-2. Examples of notifications

[0059] Notification that autonomous driving using a machine learning model is being performed may be provided in a variety of ways. Figure 4 is a conceptual diagram used to explain an example of notification. Figure 4 , the vicinity of the driver's seat of the vehicle 1 is shown. Figure 4, an instrument panel display 61 , a multi-information display 62 , a speaker 70 , and a steering wheel 2 are shown as devices provided around the driver's seat.

[0060] One example of notification is a specific display on the instrument panel display 61. Figure 4 2 shows an example of displaying a specific icon on the instrument panel display 61.

[0061] Another example of notification is a specific display on the multi-information display 62. Figure 4 , an example of making the multi-information display 62 display a specific window DP is shown. The window DP is, for example, a heat map display that represents the basis of the output of the machine learning model. In addition, as an example of a specific display in the multi-information display 62, an example of changing the color tone of the screen can be cited. In addition, when making the multi-information display 62 display the target trajectory TRJ, an example of changing the color and / or mark of the displayed target trajectory TRJ can be cited.

[0062] Another example of notification is to emit a specific sound and / or voice from the speaker 70. For example, the speaker 70 may output a voice such as "autonomous driving using AI has started".

[0063] Another example of notification is to actuate an actuator associated with the steering wheel 2. Figure 4 2 shows an example in which the steering wheel 2 is vibrated by a vibration actuator. In addition, it is also possible to change the weight of the steering wheel 2 operation by a reaction torque actuator.

[0064] By making such a notification, the operator in the driver's seat can easily know that the autonomous driving function using the machine learning model is being executed. The processor 110 may also combine multiple notifications. In addition, the processor 110 may also be configured to notify the passengers of the vehicle 1.

[0065] The processor 110 may be further configured to change the notification according to the contribution of the output of the machine learning model to the control judgment or the control amount. The contribution may be, for example, the proportion of the machine learning model in the plurality of models constituting the planning unit 30 and the control amount calculation unit 40 as an indicator.

[0066] Figure 5 (A) and (B) show an example of a case where notification is changed according to the contribution degree. Figure 5 (A) is an example in which the display of a specific icon is changed according to the contribution degree. Figure 5 (B) is an example in which the reaction torque (weight of the operation of the steering wheel 2) achieved by the reaction torque actuator is changed according to the contribution degree. Figure 5In the example shown in (B), the reaction torque increases as the contribution degree increases.

[0067] By adopting such a configuration, the operator of the vehicle 1 can further understand the general usage status of the machine learning model in the automatic driving function. As a result, the operator of the vehicle 1 can make a judgment that is consistent with the usage status of the machine learning model.

[0068] 2-3. Processing

[0069] Figure 6 1 is a flowchart showing an example of processing executed by the processor 110 regarding the above-mentioned notification to the operator of the vehicle 1 . Figure 6 The processing involved in the flowchart shown is repeatedly executed in a predetermined processing cycle, for example, while the automatic driving function is being executed, or may be executed when the model included in the recognition unit 20, the planning unit 30, or the control amount calculation unit 40 is switched.

[0070] In step S110 , the processor 110 checks the configuration of the automatic driving control unit (the recognition unit 20 , the planning unit 30 , and the control amount calculation unit 40 ).

[0071] Next, in step S120 (first process), processor 110 determines whether a machine learning model is being used.

[0072] If it is determined that the machine learning model is not used (step S120: No), the process ends without notifying the operator of the vehicle 1. If it is determined that the machine learning model is used (step S120: Yes), the process proceeds to step S130.

[0073] In step S130 , the processor 110 calculates the contribution of the output of the machine learning model.

[0074] Next, in step S140 (second process), processor 110 notifies the operator of vehicle 1 of the degree of contribution calculated in step S130. After step S140, the process ends.

[0075] 3. Effect

[0076] As described above, according to the present embodiment, it is determined whether the machine learning model is being used in the automatic driving function being executed. Furthermore, when it is determined that the machine learning model is being used, the operator of the vehicle 1 is notified that the automatic driving function using the machine learning model is being executed. Thus, it is possible to reduce the operator of the vehicle 1 from misrecognizing whether the automatic driving function using the machine learning model is being used.

[0077] In addition, the present embodiment can also be applied to other control systems installed in the vehicle 1. For example, as another control system for executing one or more control functions, a driving assistance system can be cited as an example. The driving assistance system is a system that executes control functions such as collision damage mitigation braking, cruise control, and traffic sign recognition according to the driving environment or request of the vehicle 1. When applied to the driving assistance system, it is sufficient to determine whether the machine learning model is being used in each control function being executed. Moreover, when it is determined that the machine learning model is being used in any of the control functions being executed, the operator of the vehicle 1 is notified of the fact that the control function using the machine learning model is being executed. At this time, the notification can also be changed according to the type of control function that is determined to be using the machine learning model.

Claims

1. A control system is a system mounted on a vehicle, comprising one or more processors for executing one or more control functions according to the driving environment or request of the vehicle, The one or more processors are further configured to perform: determining whether a first process using a machine learning model is being used in the control function being executed; and When it is determined that the machine learning model is being used, a second process is performed to notify the operator of the vehicle that the control function using the machine learning model is being executed.

2. The control system according to claim 1, The first process includes: When the output of the machine learning model is used as a control judgment or a control amount in the control function being executed, it is determined that the machine learning model is being utilized.

3. The control system according to claim 2, The second process includes: Calculating the contribution of the output of the machine learning model to the control judgment or control amount; and The notification is changed according to the contribution level.

4. The control system according to any one of claims 1 to 3, The first process includes: When the information of the driving environment recognized by the machine learning model is used in the control function being executed to perform control judgment or calculation of the control amount, it is determined that the machine learning model is being used.

Images