control system

Abstract

To reduce false recognition by a vehicle operator about whether a control function utilizing a machine learning model is used in an on-vehicle control system or not.SOLUTION: A control system includes one or more processors executing one or more control functions in response to a driving environment of a vehicle or a request. The one or more processors are configured so as to execute a first processing to determine whether a machine learning model is used or not in the control function under execution, and a second processing to notify a vehicle operator that the control function utilizing the machine learning model is executed when it is determined that the machine learning model is utilized.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a control system mounted on a vehicle.

Background Art

[0002] In recent years, the use of machine learning models as artificial intelligence (AI) has been promoted in various fields. Also, in each field, technologies for effectively using AI have been considered.

[0003] Patent Document 1 discloses an information processing apparatus having a user information acquisition unit that identifies a usage scene of AI by a user, and an AI selection unit that selects an AI corresponding to the identified usage scene. In addition, there is the following Patent Document 2 as a document showing the technical level of this technical field.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the future, it is expected that the use of machine learning models will also progress in the field of vehicle control. It is conceivable that a vehicle operator will use a vehicle control function using a machine learning model under a certain consent. On the other hand, various functions are implemented in vehicle control, and it is assumed that the usage status of the machine learning model differs for each control function. Therefore, there is a risk that the vehicle operator may misrecognize whether or not a control function using a machine learning model is being used. Such misrecognition may cause an unexpected situation for the vehicle operator.

[0006] One of the purposes of this disclosure is to address the above-mentioned challenges and to reduce misperceptions by vehicle operators regarding whether or not a control function utilizing a machine learning model is being used. [Means for solving the problem]

[0007] One aspect of this disclosure relates to control systems installed in vehicles.

[0008] The control system comprises one or more processors that execute one or more control functions in response to the vehicle's driving environment or requests. The one or more processors are further configured to perform a first process that determines whether or not a machine learning model is being used in the control function being executed, and a second process that, if it is determined that a machine learning model is being used, notifies the vehicle operator that a control function utilizing a machine learning model is being executed. [Effects of the Invention]

[0009] According to this disclosure, it is determined whether or not a machine learning model is being used in the control function being executed. When it is determined that a machine learning model is being used, the vehicle operator is notified that a control function utilizing the machine learning model is being executed. This reduces the likelihood of the vehicle operator misinterpreting whether or not a control function utilizing a machine learning model is being used. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example configuration related to the autonomous driving function of a vehicle according to this embodiment. [Figure 2] This figure shows an example of the configuration of the automated driving system according to this embodiment. [Figure 3] This figure shows an example of the configuration of an automated driving control unit that is determined to be using a machine learning model. [Figure 4] This is a diagram illustrating an example of a notification according to this embodiment. [Figure 5] This diagram shows an example of how notifications can be varied depending on the level of contribution. [Figure 6] This flowchart shows an example of the process performed by a processor. [Modes for carrying out the invention]

[0011] The embodiments will be described below with reference to the drawings.

[0012] 1. Autonomous driving system This embodiment relates to an automated driving system for a vehicle. The automated driving system according to this embodiment is installed in a vehicle and performs automated driving functions according to the vehicle's driving environment or requests. Figure 1 is a block diagram showing an example configuration related to the automated driving function of vehicle 1 by the automated driving system according to this embodiment. Automated driving means that at least one of the steering, acceleration, and deceleration of vehicle 1 is performed automatically without driver operation by an operator. The automated driving function is a concept that includes not only fully automated driving control but also risk avoidance control, lane keeping assist control, etc. The operator may be a driver riding in vehicle 1 or a remote operator who remotely controls vehicle 1.

[0013] Vehicle 1 includes a sensor group 10, a recognition unit 20, a planning unit 30, a control quantity calculation unit 40, and a driving device 50.

[0014] The sensor group 10 includes recognition sensors 11 used to recognize the surrounding conditions of the vehicle 1. Examples of recognition sensors 11 include cameras, LIDAR (Laser Imaging Detection and Ranging), radar, etc. The sensor group 10 may further include state sensors 12 for detecting the state of the vehicle 1, position sensors 13 for detecting the position of the vehicle 1, etc. Examples of state sensors 12 include speed sensors, acceleration sensors, yaw rate sensors, steering angle sensors, etc. An example of a position sensor 13 is a GNSS (Global Navigation Satellite System) sensor.

[0015] The sensor detection information SEN is the information obtained by the sensor group 10. For example, the sensor detection information SEN includes an image captured by a camera. As another example, the sensor detection information SEN may include point cloud information obtained by LIDAR. The sensor detection information SEN may include vehicle state information indicating the state of the vehicle 1. The sensor detection information SEN may include position information indicating the position of the vehicle 1.

[0016] The recognition unit 20 receives the sensor detection information SEN. The recognition unit 20 recognizes the driving environment of the vehicle 1 based on the information obtained by the recognition sensor 11. For example, the recognition unit 20 recognizes the objects around the vehicle 1. Examples of the objects include pedestrians, other vehicles (preceding vehicles, parked vehicles, etc.), white lines, road structures (e.g., guardrails, curbs), falling objects, traffic signals, intersections, signs, etc. Also for example, the recognition unit 20 performs self-position estimation of the vehicle 1. The recognition result information RES indicates the recognition result by the recognition unit 20. For example, the recognition result information RES includes object information indicating the relative position and relative speed of the object with respect to the vehicle 1.

[0017] The planning unit 30 receives the recognition result information RES from the recognition unit 20. Also, the planning unit 30 may receive vehicle state information, position information, and pre-generated map information. The map information may be high-precision 3D map information. The planning unit 30 generates a driving plan for the vehicle 1 based on the received information. The driving plan may be for reaching a preset destination or for avoiding risks. Examples of the driving plan include maintaining the current driving lane, changing lanes, overtaking, turning right or left, steering, accelerating, decelerating, stopping, etc. Further, the planning unit 30 generates a target trajectory TRJ necessary for the vehicle 1 to travel according to the driving plan. The target trajectory TRJ includes a target position and a target speed. The driving plan and the target trajectory TRJ generated by the planning unit 30 are, so to speak, control judgments in the autonomous driving function.

[0018] The control quantity calculation unit 40 receives the target trajectory TRJ from the planning unit 30. The control quantity calculation unit 40 calculates the control quantity CON required for the vehicle 1 to follow the target trajectory TRJ. The control quantity CON can also be described as the control quantity required to reduce the deviation between the vehicle 1 and the target trajectory TRJ. The control quantity CON includes at least one of the steering control quantity, drive control quantity, and braking control quantity. Examples of steering control quantities include target steering angle, target torque, target motor angle, target motor drive current, etc. Examples of drive control quantities include target speed, target acceleration, etc. Examples of braking control quantities include target speed, target deceleration, etc.

[0019] The running gear 50 includes a steering gear 51, a drive gear 52, and a braking gear 53. The steering gear 51 steers the wheels. For example, the steering gear 51 includes an electric power steering (EPS) system. The drive gear 52 is a power source that generates driving force. Examples of the drive gear 52 include an engine, an electric motor, an in-wheel motor, etc. The braking gear 53 generates braking force. The running gear 50 receives a control amount CON from the control amount calculation unit 40. The running gear 50 operates the steering gear 51, the drive gear 52, and the braking gear 53 according to the steering control amount, the drive control amount, and the braking control amount, respectively. As a result, the vehicle 1 travels in accordance with the target trajectory TRJ.

[0020] The recognition unit 20 includes at least one of a rule-based model and a machine learning model. The rule-based model performs recognition processing based on a predetermined set of rules. Examples of the machine learning model include NN (Neural Network), SVM (Support Vector Machine), regression models, decision tree models, etc. The NN may be a CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), or a combination thereof. The type of each layer layer, the number of layers, and the number of nodes in the NN are arbitrary. The machine learning model is generated in advance through machine learning. The recognition unit 20 performs recognition processing by inputting the sensor detection information SEN into the model. The recognition result information RES is output from the model or generated based on the output from the model. The recognition unit 20 may be configured to perform a plurality of recognition processes using a plurality of models. For example, the recognition unit 20 may be configured to include a rule-based model for self-position estimation and a machine learning model for object recognition.

[0021] Similarly, the planning unit 30 includes at least one of a rule-based model and a machine learning model. The planning unit 30 performs planning processing by inputting the recognition result information RES into the model. The target trajectory TRJ is output from the model or generated based on the output from the model. The planning unit 30 may be configured to perform planning processing using a plurality of models. For example, the planning unit 30 may be configured to include a machine learning model for generating a driving plan and a rule-based model for generating the target trajectory TRJ. Furthermore, the planning unit 30 may be configured to be able to switch the model according to the driving environment, requests, etc. of the vehicle 1. For example, the planning unit 30 may be configured to use the rule-based model while the vehicle 1 is driving on the highway and use the machine learning model while the vehicle 1 is driving on a general road in the generation of the driving plan.

[0022] Similarly, the control variable calculation unit 40 includes at least one of a rule-based model and a machine learning model. The control variable calculation unit 40 performs control variable calculation processing by inputting a target trajectory TRJ into the model. The control variable CON is output from the model or generated based on the output from the model. The control variable calculation unit 40 may be configured to perform multiple control variable calculation processes using multiple models. For example, the control variable calculation unit 40 may be configured to include a machine learning model for calculating steering control variables and a rule-based model for calculating drive control variables and braking control variables. Furthermore, the control variable calculation unit 40 may be configured to switch models according to the driving environment of vehicle 1 or requests.

[0023] Two or more of the recognition unit 20, planning unit 30, and control variable calculation unit 40 may be configured as a single unit. The recognition unit 20, planning unit 30, and control variable calculation unit 40 may all be configured as a single unit (end-to-end configuration). For example, the recognition unit 20 and planning unit 30 may be configured as a single unit by a neural network (NN) that outputs a target trajectory TRJ from sensor detection information SEN. Even in the case of a single unit configuration, intermediate products such as recognition result information RES and target trajectory TRJ may be output. For example, if the recognition unit 20 and planning unit 30 are configured as a single unit by an NN, the recognition result information RES may be the output of the intermediate layer of the NN.

[0024] The recognition unit 20, the planning unit 30, and the control quantity calculation unit 40 constitute an "automatic driving control unit" that performs the automatic driving function of the vehicle 1.

[0025] Figure 2 is a block diagram showing an example of the hardware configuration of the automated driving system 100 according to this embodiment. The automated driving system 100 has at least the functions of the automated driving control unit described above.

[0026] The autonomous driving system 100 includes a processing unit 101, a sensor group 10, a driving device 50, a display unit 60, a speaker 70, and an actuator group 80. The processing unit 101 is configured to communicate with the sensor group 10, the driving device 50, the display unit 60, the speaker 70, and the actuator group 80.

[0027] The processing unit 101 is a computer that includes one or more processors 110 (hereinafter simply referred to as processor 110) and one or more storage devices 120 (hereinafter simply referred to as storage devices 120).

[0028] The processor 110 performs various processes. Examples of the processor 110 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc. The recognition unit 20, the planning unit 30, and the control variable calculation unit 40 may be implemented by a single processor 110 or by separate processors 110. The storage device 120 stores various information. Examples of the storage device 120 include an HDD (Hard Disk Drive), an SSD (Solid State Drive), volatile memory, non-volatile memory, etc.

[0029] The storage device 120 stores the computer program 130 and model data 140.

[0030] The computer program 130 is executed by the processor 110. Various processes in the autonomous driving system 100 may be realized through the cooperation of the processor 110, which executes the computer program 130, and the storage device 120. The computer program 130 may be recorded on a computer-readable recording medium.

[0031] Model data 140 is model data included in the recognition unit 20, the planning unit 30, and the control variable calculation unit 40. Model data 140 is stored in the storage device 120. In executing the autonomous driving function, the processor 110 configures the recognition unit 20, the planning unit 30, and the control variable calculation unit 40 by selecting and using a model from the model data 140. In particular, the processor 110 may be configured to determine which model to select according to the driving environment of vehicle 1 and requests. This enables switching of models included in the recognition unit 20, the planning unit 30, and the control variable calculation unit 40.

[0032] The display unit 60 is mounted on the vehicle 1 and displays various information. The display unit 60 is configured to be controllable by the processing unit 101. Examples of the display unit 60 include a meter panel display, a multi-information display, a head-up display, an indicator, and the like.

[0033] Speaker 70 is mounted on vehicle 1 and emits various sounds. Speaker 70 is configured to be controllable by processing unit 101.

[0034] The actuator group 80 includes actuators for various devices provided by the vehicle 1. For example, the actuator group 80 includes a vibration actuator for vibrating the steering wheel and a reaction torque actuator for generating a reaction torque on the steering wheel. The actuator group 80 is configured to be controllable by the processing unit 101.

[0035] 2. Notification to the vehicle operator In the autonomous driving system 100, the autonomous driving function executed by the processor 110 may utilize machine learning models or rule-based models only. Furthermore, it is expected that the utilization of machine learning models in the recognition unit 20, the planning unit 30, and the control quantity calculation unit 40 will differ depending on the driving environment and requests of the vehicle 1.

[0036] In the automated driving system 100 according to this embodiment, the processor 110 determines whether or not a machine learning model is being used in the automated driving function that is currently running. When it is determined that a machine learning model is being used, the processor 110 notifies the operator of the vehicle 1 that an automated driving function using a machine learning model is being executed. These processes will be described below.

[0037] 2-1. Determining whether or not a machine learning model is being used. The processor 110 can determine whether or not a machine learning model is being used based on the configurations of the recognition unit 20, the planning unit 30, and the control variable calculation unit 40, from the following perspectives.

[0038] The first perspective is whether or not the output of the machine learning model is used as a control decision or control variable. In other words, according to this perspective, the processor 110 determines whether or not the machine learning model is being used based on its usage status in the planning unit 30 or the control variable calculation unit 40. For example, the processor 110 determines that the machine learning model is being used when the output of the machine learning model in the planning unit 30 is set to a driving plan or target trajectory TRJ. Also, for example, the processor 110 determines that the machine learning model is being used when the output of the machine learning model in the control variable calculation unit 40 is set to a steering control variable, a drive control variable, or a braking control variable.

[0039] Figures 3(A) and 3(B) show examples of the configuration of an automated driving control unit where a machine learning model is determined to be used from the first perspective. In Figure 3(A), the planning unit 30 is composed of a machine learning model 31 and a rule-based model 32. The output of the machine learning model 31 is the driving plan PLN. In Figure 3(B), the control amount calculation unit 40 is composed of a machine learning model 41, a machine learning model 42, and a rule-based model 43. The outputs of the machine learning models 41 and 42 are part of the control amount CON. For example, machine learning model 41 is a model for calculating the steering control amount, and machine learning model 42 is a model for calculating the drive control amount.

[0040] The second perspective is whether control decisions or control quantities are calculated using information about the driving environment of vehicle 1 recognized by the machine learning model. In other words, from this perspective, the processor 110 determines whether the machine learning model is being used based on the usage status of the recognition result information RES, which is the output of the machine learning model. For example, the processor 110 determines that the machine learning model is being used when the planning unit 30 generates a driving plan PLN or target trajectory TRJ using object information from the output of the machine learning model.

[0041] Figure 3(C) shows an example of the configuration of an automated driving control unit where a machine learning model is determined to be used from the second perspective. In Figure 3(C), the recognition unit 20 is composed of a machine learning model 21, a rule-based model 22, and a rule-based model 23. In the planning unit 30, the rule-based model 33 generates a driving plan PLN using the recognition result information RES from the output of the machine learning model 21.

[0042] As explained above, from the above perspective, the processor 110 can determine whether or not a machine learning model is being used in the running automated driving function based on the configurations of the recognition unit 20, the planning unit 30, and the control variable calculation unit 40. The processor 110 may also determine whether or not a machine learning model is being used by combining the above perspectives.

[0043] 2-2. Example of a notification Notifications indicating that autonomous driving using a machine learning model is in progress can be implemented in various forms. Figure 4 is a conceptual diagram illustrating an example of such a notification. Figure 4 shows the area around the driver's seat of vehicle 1. Figure 4 shows the instrument panel display 61, multi-information display 62, speaker 70, and steering wheel 2 as equipment installed around the driver's seat.

[0044] One example of notification is displaying a specific message on the meter panel display 61. Figure 4 shows an example of displaying a specific icon on the meter panel display 61.

[0045] Another example of notification is displaying specific information on the multi-information display 62. Figure 4 shows an example of displaying a specific window DP on the multi-information display 62. The window DP is, for example, a heat map that shows the basis for the output of a machine learning model. Other examples of specific displays on the multi-information display 62 include changing the screen's color scheme. Also, if the target trajectory TRJ is displayed on the multi-information display 62, the color and notation of the displayed target trajectory TRJ may be changed.

[0046] Another example of notification is emitting a specific sound or voice from speaker 70. For example, speaker 70 could output a voice message such as, "Starting AI-powered autonomous driving."

[0047] Another example of notification is operating an actuator related to the steering wheel 2. Figure 4 shows an example of vibrating the steering wheel 2 using a vibration actuator. Alternatively, it is conceivable to change the resistance of the steering wheel 2 using a reaction torque actuator.

[0048] By providing such notifications, the operator in the driver's seat can easily know that the automated driving function using the machine learning model is in operation. The processor 110 may combine multiple notifications. The processor 110 may also be configured to provide notifications to the occupants of vehicle 1.

[0049] The processor 110 may also be configured to change notifications according to the contribution of the machine learning model's output to the control decision or the controlled variable. The contribution can be, for example, the proportion of the machine learning model among the multiple models constituting the planning unit 30 and the controlled variable calculation unit 40.

[0050] Figures 5(A) and 5(B) show examples of how notifications can be changed according to the degree of contribution. Figure 5(A) is an example of how the display of a specific icon can be changed according to the degree of contribution. Figure 5(B) is an example of how the reaction torque (the weight of operating the steering wheel 2) by the reaction torque actuator can be changed according to the degree of contribution. In the example shown in Figure 5(B), the reaction torque increases as the degree of contribution increases.

[0051] By adopting this configuration, the operator of vehicle 1 can also learn the approximate usage status of the machine learning model in the autonomous driving function. This allows the operator of vehicle 1 to make decisions in accordance with the usage status of the machine learning model.

[0052] 2-3. Processing Figure 6 is a flowchart showing an example of the processing performed by the processor 110 in relation to the notification to the operator of vehicle 1 described above. The processing shown in the flowchart in Figure 6 is repeatedly performed at a predetermined processing cycle, for example, while the autonomous driving function is being executed. Alternatively, it may be performed when a switch occurs between models included in the recognition unit 20, the planning unit 30, or the control variable calculation unit 40.

[0053] In step S110, the processor 110 confirms the configuration of the automatic driving control unit (recognition unit 20, planning unit 30, and control amount calculation unit 40).

[0054] Next, in step S120 (first process), the processor 110 determines whether or not a machine learning model is being used.

[0055] If it is determined that the machine learning model is not being used (step S120; No), the process ends without notifying the operator of vehicle 1. If it is determined that the machine learning model is being used (step S120; Yes), the process proceeds to step S130.

[0056] In step S130, the processor 110 calculates the contribution of the machine learning model's output.

[0057] Next, in step S140 (second process), the processor 110 notifies the operator of vehicle 1 according to the contribution calculated in step S130. After step S140, the process ends.

[0058] 3. Effects As described above, according to this embodiment, it is determined whether or not a machine learning model is being used in the running automated driving function. When it is determined that a machine learning model is being used, the operator of vehicle 1 is notified that the automated driving function using the machine learning model is in operation. This reduces the likelihood of the operator of vehicle 1 misinterpreting whether or not the automated driving function using the machine learning model is being used.

[0059] This embodiment can also be applied to other control systems installed in Vehicle 1. For example, a driver assistance system can be considered as another control system for executing one or more control functions. The driver assistance system is a system that executes control functions such as collision mitigation braking, cruise control, and traffic sign recognition in response to the driving environment or requests of Vehicle 1. When applied to a driver assistance system, it is sufficient to determine whether or not a machine learning model is being used in each control function that is currently being executed. When it is determined that a machine learning model is being used in any of the currently being executed control functions, the operator of Vehicle 1 should be notified that a control function using a machine learning model is being executed. At this time, the notification may be changed depending on the type of control function that is determined to be using a machine learning model. [Explanation of symbols]

[0060] 1 Vehicle, 10 Sensor Groups, 20 Recognition Unit, 30 Planning Unit, 40 Control Quantity Calculation Unit, 50 Traveling device, 60 Display unit, 70 Speaker, 80 Actuator group, 100 autonomous driving systems, 110 processors, 120 memory devices, 130 computer programs, 140 model data

Claims

1. A control system installed in a vehicle, The vehicle comprises one or more processors that perform one or more control functions in response to the vehicle's operating environment or requests. The one or more processors further include: The first process determines whether or not a machine learning model is being used in the control function being executed, When it is determined that a machine learning model is being used, a second process is performed to notify the operator of the vehicle that a control function using the machine learning model is in operation. It is configured to execute Control system.

2. A control system according to claim 1, The first process includes determining that a machine learning model is being used when the output of the machine learning model is being used as a control decision or control variable in the control function being executed. Control system.

3. A control system according to claim 2, The second process described above is: A process for calculating the contribution of the output of the machine learning model to the control judgment or controlled variable, The notification will be changed according to the aforementioned level of contribution, including Control system.

4. A control system according to any one of claims 1 to 3, The first process includes determining that a machine learning model is being used when a control decision or calculation of a control amount is being performed using information about the operating environment recognized by the machine learning model in the control function that is currently being executed. Control system.

Images