Autonomous Driving System

Abstract

To provide an automatic operation system which allows a user to monitor automatic operation by a machine learning model properly.SOLUTION: An automatic operation system includes: an external sensor; an internal sensor which detects a running state of a vehicle; a machine learning model which outputs an indicated value of automatic operation of the vehicle by using a detection result of the external sensor and a detection result of the internal sensor as input values; an operation part which automatically operates the vehicle based on the indicated value output by the machine learning model; a user interface which receives / provides information from / to an occupant of the vehicle; and a control unit. The control unit determines at least one risk candidate scene based on at least one of the detection result of the external sensor, the detection result of the internal sensor, and operation states of the occupant of the vehicle, extracts the input values and the indicated value of the machine learning model in the at least one risk candidate scene, and causes the user interface to provide monitoring information including extraction of the input values and the indicated value to the occupant of the vehicle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to automated driving systems. [Background technology]

[0002] Patent Document 1 discloses an apparatus for collecting training data for machine learning for autonomous driving. The apparatus includes a sensor that acquires external environment information and a control unit that assigns a label to information of interest included in the acquired external environment information. If the vehicle can pass through without taking evasive action at the time the external environment information is collected, the control unit assigns a label indicating passable to the information of interest. If the vehicle takes evasive action, the control unit assigns a label indicating impassable to the information of interest. This automates at least a portion of the labeling (annotation work) of the external environment information, thereby reducing the effort, time, and cost of generating training data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 116423 Summary of the Invention [Problem to be solved by the invention]

[0004] A trained model generated by machine learning (hereinafter referred to as a machine learning model) may not be able to achieve appropriate vehicle control in all situations. Therefore, in order to properly operate autonomous driving control using a machine learning model, a configuration for monitoring the autonomous driving control is essential. The present disclosure provides a technology that allows a user to properly monitor autonomous driving using a machine learning model. [Means for solving the problem]

[0005] An autonomous driving system according to one embodiment of the present disclosure includes an external sensor, an internal sensor, a machine learning model, a driving unit, a user interface, and a control unit. The external sensor detects the external environment of the vehicle. The internal sensor detects the driving state of the vehicle. The machine learning model uses the detection results of the external sensor and the internal sensor as input values ​​and outputs instruction values ​​for autonomous driving of the vehicle. The driving unit automatically drives the vehicle based on the instruction values ​​output by the machine learning model. The user interface exchanges information with vehicle occupants. The control unit is connected to the user interface. The control unit determines at least one risk candidate scene based on at least one of the detection results of the external sensor, the detection results of the internal sensor, and the driving state of the vehicle occupant. The control unit extracts input values ​​and instruction values ​​for the machine learning model in the at least one risk candidate scene. The control unit causes the user interface to notify the vehicle occupant of monitoring information including information indicating that the input values ​​and instruction values ​​have been extracted.

[0006] In this autonomous driving system, at least one risk candidate scene is determined based on at least one of the detection results of the external sensors, the detection results of the internal sensors, and the driving state of the vehicle occupants. Then, input values ​​and instruction values ​​of the machine learning model for the at least one risk candidate scene are extracted. Then, monitoring information including information indicating that the input values ​​and instruction values ​​have been extracted is notified to the vehicle occupants. In this way, the autonomous driving system can automatically determine the risk candidate scene and notify the user (vehicle occupant) of the monitoring information, allowing the user to appropriately monitor autonomous driving using the machine learning model.

[0007] In one embodiment, the autonomous driving system may further include a transmission unit that transmits the monitoring information to a device outside the vehicle. The monitoring information may include request information that allows the user to select whether to transmit the monitoring information to the device outside the vehicle. In response to receiving a response permitting transmission to the device outside the vehicle via the user interface, the control unit may cause the transmission unit to transmit extracted data, which is the input values ​​and instruction values ​​of the extracted machine learning model, to the device outside the vehicle. Since information is transmitted (uploaded) based on user permission, the autonomous driving system can realize user-led monitoring and notification.

[0008] In one embodiment, the control unit may present at least one risk candidate scene to the vehicle occupant via a user interface, and cause the transmission unit to transmit extracted data corresponding to the risk candidate scene selected by the vehicle occupant via the user interface. Since information is transmitted (uploaded) based on the user selection, the autonomous driving system can realize user-driven monitoring and notification.

[0009] In one embodiment, the control unit may receive a ranking of at least one risk candidate scene designated by a vehicle occupant via a user interface, and determine the extracted data to be transmitted by the transmission unit based on the ranking. Since information is transmitted (uploaded) based on the user's designation, the automated driving system can realize user-driven monitoring and notification. Furthermore, the automated driving system can prioritize the extracted data to be uploaded.

[0010] In one embodiment, the control unit may calculate a similarity between the input value and the instruction value in at least one risk candidate scene, classify the at least one risk candidate scene into categories based on the calculated similarity, and present at least one risk candidate scene for each of the categorized categories to the vehicle occupant via a user interface. Since the risk candidate scenes are presented for each category, the autonomous driving system can present information to the user to make it easier for the user to select a risk candidate scene.

[0011] In one embodiment, the control unit may extract data such that extracted data corresponding to a risk candidate scene for which an instruction that the risk candidate scene is a risk candidate scene is received from a vehicle occupant via a user interface contains more information than extracted data corresponding to a risk candidate scene for which no instruction is received. Because the extracted data corresponding to the user-specified risk candidate scene contains more information, the automated driving system can realize user-driven monitoring and notification. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to allow a user to appropriately monitor automated driving using a machine learning model. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a vehicle equipped with an autonomous driving system according to an embodiment. [Figure 2] FIG. 2 is a flowchart illustrating an example of the operation of the autonomous driving system according to an embodiment. [Figure 3] FIG. 3 is a flowchart illustrating an example of the operation of the autonomous driving system according to an embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of the operation of the autonomous driving system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated description will be omitted.

[0015] [Vehicle configuration] Fig. 1 is a block diagram showing an example of the configuration of a vehicle equipped with an autonomous driving system according to an embodiment. As shown in Fig. 1, the autonomous driving system 1 is mounted on a vehicle 2 as an example, and causes the vehicle 2 to function as an autonomously driven vehicle. The autonomous driving system 1 operates actuators 3 of the vehicle 2 to achieve autonomous driving. The actuators 3 include, for example, an engine actuator, a brake actuator, and a steering actuator.

[0016] The autonomous driving system 1 includes, as an example, an external sensor 10, an internal sensor 11, an autonomous driving ECU 12, a user interface 13, a driving state detection sensor 14, a control unit 15, a memory unit 16, and a transmission unit 17.

[0017] The external sensor 10 is an on-board sensor that detects the external environment of the vehicle 2. The external sensor 10 includes at least either a camera or a radar sensor. The external sensor 10 outputs images and detection information related to surrounding objects to the autonomous driving ECU 12.

[0018] The internal sensor 11 is an on-board sensor that detects the driving state of the vehicle 2. The internal sensor 11 includes, for example, a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The internal sensor 11 outputs detection information related to the driving state of the vehicle 2 to the automatic driving ECU 12.

[0019] The autonomous driving ECU 12 includes a machine learning model 120 and a driving unit 121. The ECU (Electronic Control Unit) is an electronic control unit that includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CAN (Controller Area Network) communication circuit, etc. The autonomous driving ECU 12 uses the machine learning model 120 to realize autonomous driving using an AI (Artificial Intelligence) system.

[0020] The machine learning model 120 receives the detection results of the external sensor 10 and the internal sensor 11 as input values ​​and outputs instruction values ​​for autonomous driving of the vehicle 2. One example of the machine learning model 120 is a recursive deep learning model. The machine learning model 120 learns based on teacher data including, for example, input values ​​and instruction values ​​for autonomous driving, and distributes the learned data to the vehicle 2. The instruction values ​​for autonomous driving are control values ​​for the actuator 3. The driving unit 121 operates the actuator 3 based on the instruction values ​​output by the machine learning model 120, and drives the vehicle 2 automatically.

[0021] The user interface 13 is a device that exchanges information with the occupants of the vehicle 2. Hereinafter, the occupants will also be referred to as users. The user interface 13 receives monitoring information from the control unit 15 and notifies the occupants. The monitoring information will be described later. The user interface 13 accepts user operations from the occupants. One example of the user interface 13 is a touch panel. The user interface 13 may include a speaker and a microphone. In other words, notifications include audio notifications, and user operations include audio instructions.

[0022] The driving state detection sensor 14 is a device that detects the driving state of the occupant. The driving state detection sensor 14 detects an intervention into automatic driving by the occupant (driver intervention), or a sudden acceleration operation, a sudden braking operation, or a sudden steering operation. The driving state detection sensor 14 may be, for example, a brake pedal sensor, an accelerator pedal sensor, or a steering sensor. The driving state detection sensor 14 may include a driver monitor camera or a biosensor and detect the driving state of the occupant.

[0023] The control unit 15 is a device that controls operations for monitoring autonomous driving. The control unit 15 is configured by, for example, an ECU. The control unit 15 acquires input values ​​and instruction values ​​of the machine learning model 120 and stores them in the memory unit 16. The memory unit 16 is a storage device such as an HDD (Hard Disk Drive). The memory unit 16 stores the input values ​​and instruction values ​​of the machine learning model 120 for a predetermined period of time.

[0024] The control unit 15 determines at least one risk candidate scene based on at least one of the detection results of the external sensor 10, the detection results of the internal sensor 11, and the driving state of the occupants of the vehicle 2. Examples of risk candidate scenes include scenes in which driver intervention (accelerator, brake, etc.) occurs, sudden acceleration / deceleration occurs, the steering angle is equal to or greater than a threshold, the distance between the vehicle 2 and surrounding vehicles is equal to or less than a threshold, the distance between the vehicle 2 and surrounding lane lines is equal to or less than a threshold, the horn is honked, the driver behaves in a surprised manner, etc.

[0025] The control unit 15 may determine as risk candidate scenes a scene in which a lane departure risk has occurred and a scene in which a collision risk between the vehicle 2 and another object has occurred. For example, the control unit 15 uses the lane position relative to the vehicle 2, the positions of surrounding traffic participants, and the recognition results of surrounding structures to determine as a scene in which a lane departure risk has occurred the first scene in which the lane boundary line from the center of the vehicle 2 is below a threshold or is at its smallest in a situation in which there are no surrounding traffic participants or surrounding structures. The control unit 15 determines as a scene in which a collision risk with another object has occurred the first scene in which the distance between the predicted positions of the vehicle 2 and other traffic participants at each time is below a threshold or is at its smallest. By adopting the minimum value, the control unit 15 can determine risk candidate scenes without relying on the differences in perceived risk among users, which is difficult to set in advance.

[0026] The control unit 15 may be connected to the user interface 13, acquire the occupant's voice via the user interface 13, and determine a risk candidate scene based on the voice recognition results. For example, the control unit 15 may recognize occupant statements such as "The response to the previous vehicle that cut in was slow and dangerous" or "I can't see any reason why I'm driving toward the right of the lane," and determine a scene corresponding to the statement as a risk candidate scene. For example, based on the statement "The response to the previous vehicle that cut in was slow and dangerous," the control unit 15 determines a scene in which "a vehicle traveling in another lane is entering the vehicle's own lane" at the most recent time in data tracing back from the time of the statement as a risk candidate scene. The control unit 15 may then automatically identify data from the time the vehicle began to move toward the vehicle's own lane to the time of the statement. Alternatively, based on the statement "I can't see any reason why I'm driving toward the right of the lane," the control unit 15 determines a scene in which "the vehicle is approaching the right side of the lane" at the most recent time in data tracing back from the time of the statement as a risk candidate scene. Then, data from the time when the vehicle 2 was traveling in the center of the lane before starting that behavior to the time when the statement was made may be automatically identified.

[0027] The control unit 15 extracts input values ​​and instruction values ​​of the machine learning model 120 for at least one risk candidate scene. The control unit 15 refers to the storage unit 16, extracts input values ​​and instruction values ​​of the machine learning model 120 that match the time range in which the risk candidate scene occurred, and sets these as extracted data D. The time range in which the risk candidate scene occurred may be a predetermined time before or after the timing of the occurrence of the risk candidate scene.

[0028] The control unit 15 notifies the occupants of the vehicle 2 of the monitoring information via the user interface 13. The monitoring information is information for monitoring the autonomous driving using the machine learning model 120, and includes information that the input values ​​and instruction values ​​of the machine learning model 120 in the risk candidate scene have been extracted.

[0029] The control unit 15 causes the transmission unit 17 to transmit the extracted data D to the server 4. The transmission unit 17 is a communication device capable of wireless communication. This uploads only the extracted data. The uploaded information can be used for early risk detection and response.

[0030] The control unit 15 may cause the transmission unit 17 to transmit the extracted data D to the server 4 only with the permission of an occupant of the vehicle 2. For example, the monitoring information includes request information that prompts the occupant of the vehicle 2 to select whether or not to transmit the extracted data D to the server 4. The control unit 15 presents the request information to the occupant of the vehicle 2 along with a notice that the extracted data D has been extracted. The control unit 15 causes the transmission unit 17 to transmit the extracted data D to the server 4 in response to receiving a response permitting transmission to the server 4 via the user interface. In this way, the extracted data D may be uploaded only with the user's permission. In this case, only permitted information can be uploaded, thereby reducing communication resources and avoiding data capacity congestion.

[0031] The control unit 15 may present at least one risk candidate scene to the occupant of the vehicle 2 via the user interface 13, and may cause the transmission unit 17 to transmit extracted data D corresponding to the risk candidate scene selected by the occupant of the vehicle 2 via the user interface 13. The control unit 15 presents the risk candidate scenes that occurred within a predetermined period to the occupant of the vehicle 2. The control unit 15 may present categorized risk candidate scenes. For example, the control unit 15 may calculate the similarity between the input values ​​and the instruction values ​​in the risk candidate scene, and classify the risk candidate scene into categories based on the calculated similarity. This allows the control unit 15 to present the risk candidate scenes to the occupant of the vehicle 2 for each classified category.

[0032] The occupant of the vehicle 2 selects a risk candidate scene to be uploaded from the presented risk candidate scenes. The user interface 13 accepts the occupant's selection operation. The control unit 15 generates extracted data D corresponding to the selected risk candidate scene. The control unit 15 then transmits the extracted data D to the transmission unit 17. In this case, necessary information can be selected and uploaded, thereby reducing communication resources and avoiding data capacity pressure.

[0033] The control unit 15 may receive a ranking of at least one risk candidate scene designated by an occupant of the vehicle 2 via the user interface 13, and determine the extracted data D to be transmitted by the transmission unit 17 based on the ranking. For example, the occupant of the vehicle 2 may designate in advance the priority of the above-mentioned risk candidate scenes as a ranking. When multiple risk candidate scenes have been determined, the control unit 15 determines the risk candidate scene based on the ranking and generates the extracted data D. This allows the control unit 15 to generate extracted data D corresponding to a high-ranking (high-priority) risk candidate scene when there is a limit on the communication volume.

[0034] The control unit 15 may change the size of the extracted data D depending on the risk candidate scene. For example, the control unit 15 may set a time range for extracting the extracted data D for each risk candidate scene. Alternatively, when the control unit 15 is transmitting automatically extracted extracted data D with comprehensive transmission permitted, and the control unit 15 receives an instruction from an occupant of the vehicle 2 via the user interface 13 that a certain scene is a risk candidate scene, the control unit 15 may extract data that includes more information than the extracted data D corresponding to the risk candidate scene for which the control unit 15 has not received an instruction.

[0035] Next, an operation of the autonomous driving system 1 will be described. Fig. 2 is a flowchart showing an example of the operation of the autonomous driving system according to one embodiment. The flowchart shown in Fig. 2 starts when the autonomous driving system 1 receives a start instruction operation during autonomous driving by the autonomous driving system 1.

[0036] As shown in FIG. 2, first, in step S10, the control unit 15 of the autonomous driving system 1 determines whether or not there is a risk candidate scene. The control unit 15 determines that there is a risk candidate scene when it detects driver intervention (accelerator, brake, etc.), sudden acceleration / deceleration, a steering angle equal to or greater than a threshold, a distance between the vehicle 2 and a surrounding vehicle equal to or less than a threshold, a distance between the vehicle 2 and a surrounding lane marking equal to or less than a threshold, the horn being honked, or the driver being startled. If it is determined that there is a risk candidate scene (step S10: YES), the control unit 15 extracts input / output information of the machine learning model 120 in step S12. This generates extracted data D. Then, in step S14, the control unit 15 notifies the user of the monitoring information. The control unit 15 displays the monitoring information, including the fact that the extracted data D has been generated, via the user interface 13, or notifies the user by voice. If step S14 is completed or if it is determined that there is no risk candidate scene (step S10: NO), the flowchart shown in FIG. 2 ends. By executing the flowchart shown in FIG. 2, the user can appropriately monitor the automated driving performed by the machine learning model 120.

[0037] Next, another operation of the autonomous driving system 1 will be described. Fig. 3 is a flowchart showing an example of the operation of the autonomous driving system according to one embodiment. The flowchart shown in Fig. 3 starts when the autonomous driving system 1 receives a start instruction operation during autonomous driving by the autonomous driving system 1.

[0038] Steps S20, S22, and S24 shown in FIG. 3 are the same as steps S10, S12, and S14 shown in FIG.

[0039] When step S24 is completed, the control unit 15 determines, in step S26, whether or not the user has given permission to transmit the extracted data D. For example, the control unit 15 displays a transmission permission button or the like as request information together with the monitoring information, and prompts the user to select the transmission permission button. If it is determined that the user has given permission (step S26: YES), the control unit 15 operates the transmission unit 17 to upload the extracted data D to the server 4 in step S28. When step S28 is completed, if it is determined that there is no risk candidate scene (step S20: NO), or if it is determined that the user has not given permission (step S26: NO), the flowchart shown in FIG. 3 ends. By executing the flowchart shown in FIG. 3, the extracted data D can be uploaded only when the user has given permission.

[0040] Next, a description will be given of another operation of the autonomous driving system 1. Fig. 4 is a flowchart showing an example of the operation of the autonomous driving system according to one embodiment. The flowchart shown in Fig. 4 starts when the autonomous driving system 1 receives a start instruction operation during autonomous driving by the autonomous driving system 1.

[0041] Steps S30, S32, and S34 shown in Fig. 4 are the same as steps S10, S12, and S14 shown in Fig. 2. Step S36 shown in Fig. 4 is the same as step S26 shown in Fig. 3.

[0042] If it is determined that the user has given permission (step S36: YES), the control unit 15 determines, at step S38, whether or not extracted data D has been selected. For example, the control unit 15 presents risk candidate scenes to the user via the user interface 13. The user interface 13 accepts a user selection operation of a risk candidate scene. If it is determined that a risk candidate scene has been selected (step S38: YES), the control unit 15 extracts data corresponding to the selected risk candidate scene and generates extracted data D, at step S40. Then, the control unit 15 operates the transmission unit 17 to upload the extracted data D to the server 4.

[0043] If it is determined that no risk candidate scene has been selected (step S38: NO), then in step S42, the control unit 15 extracts data corresponding to all of the presented risk candidate scenes and generates extracted data D. Then, the control unit 15 operates the transmission unit 17 to upload the extracted data D to the server 4.

[0044] When steps S40 and S42 are completed, when it is determined that there are no risk candidate scenes (step S30: NO), or when it is determined that there is no user permission (step S36: NO), the flowchart shown in Fig. 4 ends. By executing the flowchart shown in Fig. 4, the extracted data D related to the risk candidate scenes selected by the user can be uploaded.

[0045] [Summary of the embodiment] According to the autonomous driving system 1, at least one risk candidate scene is determined based on at least one of the detection results of the external sensor 10, the detection results of the internal sensor 11, and the driving state of the occupants of the vehicle 2. Then, input values ​​and instruction values ​​of the machine learning model 120 in the at least one risk candidate scene are extracted. Then, monitoring information including the fact that the input values ​​and instruction values ​​have been extracted is notified to the occupants of the vehicle 2. In this way, the autonomous driving system 1 can automatically determine risk candidate scenes and notify the user (vehicle occupant) of the monitoring information, allowing the user to appropriately monitor autonomous driving by the machine learning model 120.

[0046] Although exemplary embodiments have been described above, various omissions, substitutions, and modifications may be made without being limited to the above-described exemplary embodiments.

[0047] For example, the passenger's permission to upload does not need to be given every time monitoring information is notified, and may be given comprehensively when the autonomous driving system 1 is started up.

[0048] The server 4 may be configured as a server group. The server 4 may include a data server that compiles data uploaded from each vehicle. The data server may aggregate information from each vehicle and transmit the information via a web page or the like. This allows a ranking of risk candidate scenes that takes into account the information of each vehicle to be posted on the web page. The user's permission or selection in the above-described embodiment may be performed via the posted web page. A system may be introduced in which a occupant who selects a scene that many occupants select as a risk candidate scene can receive rewards such as points. [Explanation of symbols]

[0049] 1...Autonomous driving system, 2...Vehicle, 10...External sensor, 11...Internal sensor, 13...User interface, 15...Control unit, 17...Transmission unit, 120...Machine learning model, 121...Driving unit

Claims

1. an external sensor for detecting an external environment of the vehicle; an internal sensor for detecting the running state of the vehicle; a machine learning model that receives the detection results of the external sensors and the detection results of the internal sensors as input values ​​and outputs instruction values ​​for autonomous driving of the vehicle; a driving unit that automatically drives the vehicle based on the instruction value output by the machine learning model; a user interface for interacting with an occupant of the vehicle; a control unit connected to the user interface; Equipped with The control unit determining at least one risk candidate scene based on at least one of the detection results of the external sensor, the detection results of the internal sensor, and a driving state of an occupant of the vehicle; extracting the input values ​​and the instruction values ​​of the machine learning model for the at least one risk candidate scene; causing the user interface to notify a vehicle occupant of monitoring information including a notification that the input value and the instruction value have been extracted. Autonomous driving system.

2. a transmitter that transmits the monitoring information to a device outside the vehicle; the monitoring information includes request information for allowing a user to select whether or not to transmit the monitoring information to a device outside the vehicle; the control unit, in response to receiving a response permitting transmission to a device outside the vehicle via the user interface, causes the transmission unit to transmit extracted data, which is the input values ​​and the instruction values ​​of the extracted machine learning model, to the device outside the vehicle. The automated driving system according to claim 1 .

3. The autonomous driving system of claim 2, wherein the control unit presents the at least one risk candidate scene to an occupant of the vehicle via the user interface and causes the transmission unit to transmit the extracted data corresponding to the risk candidate scene selected by the occupant of the vehicle via the user interface.

4. 4. The autonomous driving system according to claim 3, wherein the control unit receives a ranking of the at least one risk candidate scene specified by an occupant of the vehicle via the user interface, and determines the extracted data to be transmitted by the transmission unit based on the ranking.

5. The control unit Calculating a similarity between the input value and the indication value in the at least one risk candidate scene; classifying the at least one risk candidate scene into a category based on the calculated similarity; The automated driving system of claim 3 , wherein at least one risk candidate scene is presented to the vehicle occupant for each of the classified categories via the user interface.

6. The control unit The autonomous driving system of claim 3, wherein data is extracted so that extracted data corresponding to a risk candidate scene for which an instruction that the scene is a risk candidate scene is received from an occupant of the vehicle via the user interface contains more information than extracted data corresponding to a risk candidate scene for which no instruction is received.

Images