Automatic driving system

Abstract

The present disclosure provides an automatic driving system that allows a user to appropriately monitor automatic driving implemented by a machine learning model. The automatic driving system includes: an external sensor; an internal sensor for detecting the traveling state of the vehicle; a machine learning model that outputs an instruction value for automatic driving of the vehicle using a detection result of the external sensor and a detection result of the internal sensor as input values; a driving unit that automatically drives the vehicle on the basis of the instruction value output by the machine learning model; a user interface exchanging information with an occupant of the vehicle; and a control unit that determines at least one risk candidate scene on the basis of at least one of a detection result of the external sensor, a detection result of the internal sensor, and a driving state of an occupant of the vehicle, extracts an input value and an instruction value of a machine learning model in the at least one risk candidate scene, and causes the user interface to report, to the occupant of the vehicle, monitoring information including the extracted input value and instruction value.

Description

Technical Field

[0001] The present disclosure relates to automated driving systems. Background Art

[0002] Patent document 1 discloses a device for collecting training data for machine learning for autonomous driving. The device comprises: a sensor for acquiring external environment information; and a control unit for assigning labels to information of interest contained in the acquired external environment information. In a case where the vehicle can pass without performing avoidance action at the time when the external environment information is collected, the control unit assigns a label indicating that the vehicle can pass to the information of interest. In a case where an avoidance action is performed, the control unit assigns a label indicating that the vehicle cannot pass to the information of interest. As a result, at least a portion of the labeling (annotation operation) of the external environment information is automated, thereby achieving a reduction in labor, time, and cost of the training data generation operation.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2019 / 116423

[0006] The trained model generated by machine learning (hereinafter referred to as the machine learning model) does not achieve appropriate vehicle control in all scenarios. Therefore, in order to properly use the autonomous driving control achieved by the machine learning model, a structure for monitoring the autonomous driving control is essential. Summary of the invention

[0007] The present disclosure provides technology that enables a user to appropriately monitor autonomous driving achieved through a machine learning model.

[0008] An autonomous driving system of one embodiment of the present disclosure includes an external sensor, an internal sensor, a machine learning model, a driving unit, a user interface, and a control unit. The external sensor detects the external environment of the vehicle. The internal sensor detects the driving state of the vehicle. The machine learning model uses the detection results of the external sensor and the detection results of the internal sensor as input values ​​and outputs an indication value of the autonomous driving of the vehicle. The driving unit automatically drives the vehicle based on the indication value output by the machine learning model. The user interface exchanges information with the passengers of the vehicle. The control unit is connected to the user interface. The control unit determines at least one risk candidate scenario based on at least one of the detection results of the external sensor, the detection results of the internal sensor, and the driving state of the passengers of the vehicle. The control unit extracts the input value and indication value of the machine learning model in at least one risk candidate scenario. The control unit causes the user interface to report monitoring information to the passengers of the vehicle containing the extracted input value and indication value.

[0009] In the autonomous driving system, at least one risk candidate scenario is determined based on at least one of the detection results of external sensors, the detection results of internal sensors, and the driving state of the vehicle's passengers. Then, the input value and the indication value of the machine learning model in at least one risk candidate scenario are extracted. Then, monitoring information containing the extracted input value and indication value is reported to the vehicle's passengers. In this way, the autonomous driving system can automatically determine the risk candidate scenario and report the monitoring information to the user (the vehicle's passenger), thereby enabling the user to appropriately monitor the autonomous driving achieved by the machine learning model.

[0010] In one embodiment, the autonomous driving system may further include: a sending unit that sends monitoring information to a device outside the vehicle. The monitoring information may include request information for selecting whether to permit transmission to a device outside the vehicle. The control unit may cause the sending unit to send extracted data to a device outside the vehicle in response to receiving a reply to permit transmission to a device outside the vehicle via a user interface, wherein the extracted data is an input value and an indication value of an extracted machine learning model. The information is sent (uploaded) based on user permission, so that the autonomous driving system can enable the user as the subject to monitor and notify.

[0011] In one embodiment, the control unit may present at least one risk candidate scenario to a passenger of the vehicle via a user interface, and cause the sending unit to send the extracted data corresponding to the selected risk candidate scenario to the passenger of the vehicle via the user interface. The information is sent (uploaded) based on the user selection, so that the autonomous driving system can enable the user to monitor and notify as the subject.

[0012] In one embodiment, the control unit may receive a ranking of at least one risk candidate scenario specified by a passenger of the vehicle via a user interface, and determine the extracted data to be sent by the sending unit based on the ranking. Information is sent (uploaded) based on the user's specification, so that the autonomous driving system can monitor and notify the user as a subject. In addition, the autonomous driving system can give priority to the uploaded extracted data.

[0013] In one embodiment, the control unit may calculate the similarity between the risk candidate scenarios based on the input value and the indication value in the risk candidate scenario, classify the risk candidate scenarios into categories based on the calculated similarity, and present the risk candidate scenarios to the occupants of the vehicle via the user interface according to each of the categories. The risk candidate scenarios are presented according to each category, so that the automatic driving system can present information to the user in a manner that makes it easy for the user to select the risk candidate scenario.

[0014] In one embodiment, the control unit may extract data in such a manner that the extracted data corresponding to a risk candidate scene for which an indication that the scene is a risk candidate has been received from a passenger of the vehicle via a user interface contains more information than the extracted data corresponding to a risk candidate scene for which the indication has not been received. The extracted data corresponding to the risk candidate scene designated by the user contains more information, so that the autonomous driving system can enable the user to perform monitoring and notification as a subject.

[0015] Effects of the Invention

[0016] According to the present disclosure, users can appropriately monitor autonomous driving achieved through machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a block diagram showing an example of the configuration of a vehicle including an automatic driving system according to an embodiment.

[0018] Figure 2 This is a flowchart showing an example of the operation of an autonomous driving system according to one embodiment.

[0019] Figure 3 This is a flowchart showing an example of the operation of an autonomous driving system according to one embodiment.

[0020] Figure 4 This is a flowchart showing an example of the operation of an autonomous driving system according to one embodiment.

[0021] Description of reference numerals:

[0022] 1: Autonomous driving system; 2: Vehicle; 10: External sensor; 11: Internal sensor; 13: User interface; 15: Control unit; 17: Sending unit; 120: Machine learning model; 121: Driving unit. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and repeated descriptions are omitted.

[0024] [Vehicle configuration]

[0025] Figure 1 FIG. 1 is a block diagram showing an example of a configuration of a vehicle having an automatic driving system according to an embodiment. Figure 1 As shown, as an example, an automatic driving system 1 is mounted on a vehicle 2, so that the vehicle 2 functions as an automatic driving vehicle. The automatic driving system 1 operates an actuator 3 of the vehicle 2 to realize automatic driving. The actuator 3 includes, for example, an engine actuator, a brake actuator, and a steering actuator.

[0026] As an example, the automatic driving system 1 includes an external sensor 10 , an internal sensor 11 , an automatic driving ECU 12 , a user interface 13 , a driving state detection sensor 14 , a control unit 15 , a storage unit 16 , and a transmission unit 17 .

[0027] The external sensor 10 is a vehicle-mounted sensor that detects the external environment of the vehicle 2. The external sensor 10 includes at least one of a camera and a radar sensor. The external sensor 10 outputs detection information related to an image and surrounding objects to the automatic driving ECU 12.

[0028] The internal sensor 11 is a vehicle-mounted sensor that detects the running state of the vehicle 2. The internal sensor 11 includes, for example, a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The internal sensor 11 outputs detection information related to the running state of the vehicle 2 to the automatic driving ECU 12.

[0029] The automatic driving ECU 12 includes a machine learning model 120 and a driving unit 121. The ECU (Electronic Control Unit) is an electronic control unit having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CAN (Controller Area Network) communication circuit, etc. The automatic driving ECU 12 uses the machine learning model 120 to realize automatic driving based on an AI (Artificial Intelligence) system.

[0030] The machine learning model 120 uses the detection results of the external sensor 10 and the detection results of the internal sensor 11 as input values, and outputs an indication value for the automatic driving of the vehicle 2. As an example, the machine learning model 120 is a recursive deep learning model. The machine learning model 120 learns based on training data including input values ​​and indication values ​​for automatic driving, and distributes it to the vehicle 2. The indication value for automatic driving is a control value of the actuator 3. The driving unit 121 operates the actuator 3 based on the indication value output by the machine learning model 120, and automatically drives the vehicle 2.

[0031] The user interface 13 is a device for exchanging information with passengers of the vehicle 2. Hereinafter, passengers are also referred to as users. The user interface 13 receives monitoring information from the control unit 15 and reports it to the passengers. The monitoring information will be described later. The user interface 13 accepts user operations from the passengers. As an example, the user interface 13 is a touch panel. The user interface 13 may also include a speaker and a microphone. That is, the report also includes a report by voice, and the user operation also includes instructions by voice.

[0032] The driving state detection sensor 14 is a device for detecting the driving state of the passenger. The driving state detection sensor 14 detects the passenger's intervention operation (driver intervention) to the automatic driving, or sudden acceleration operation, sudden braking operation or sudden steering. The driving state detection sensor 14 may be, for example, a brake pedal sensor, an accelerator pedal sensor, or a steering sensor. Alternatively, the driving state detection sensor 14 may include a driver monitoring camera or a biosensor to detect the passenger's driving state.

[0033] The control unit 15 is a device that manages the actions for monitoring the automatic driving. The control unit 15 is composed of, for example, an ECU. The control unit 15 obtains the input value and the indication value of the machine learning model 120 and stores them in the storage unit 16. The storage unit 16 is a storage device such as a HDD (Hard Disk Drive). The storage unit 16 stores the input value and the indication value of the machine learning model 120 within a specified period.

[0034] The control unit 15 determines at least one risk candidate scenario based on at least one of the detection results of the external sensor 10, the detection results of the internal sensor 11, and the driving state of the occupant of the vehicle 2. As an example, the risk candidate scenario is a scenario in which driver intervention (accelerator, brake, etc.), sudden acceleration and deceleration, steering angle greater than or equal to a threshold, the distance between the vehicle 2 and surrounding vehicles is less than or equal to a threshold, the distance between the vehicle 2 and the surrounding dividing line is less than or equal to a threshold, the horn is honked, the driver is startled, and the like occurs.

[0035] The control unit 15 may also set a scenario in which a risk of deviation from the driving road is generated and a scenario in which a risk of collision between the vehicle 2 and other objects is generated as a risk candidate scenario. For example, the control unit 15 uses the position of the lane relative to the vehicle 2, the positions of the surrounding traffic participants, and the recognition results of the surrounding structures, and in the case where there are no traffic participants or surrounding structures around, sets the initial scenario in which the lane boundary line is below a threshold or is the minimum from the center of the vehicle 2 as a scenario in which a risk of deviation from the driving road is generated. The control unit 15 sets the initial scenario in which the distance between the predicted positions of the vehicle 2 and other traffic participants at each moment is below a threshold or is the minimum as a scenario in which a risk of collision with other objects is generated. The control unit 15 adopts the minimum value, thereby being able to determine the risk candidate scenario without relying on differences that are difficult to set in advance and that make each user feel risky.

[0036] Alternatively, the control unit 15 is connected to the user interface 13, obtains the voice of the passenger via the user interface 13, and determines the risk candidate scene based on the voice recognition result. For example, the control unit 15 may recognize the passenger's speech such as "the response to the cutting-in vehicle just now is slow and dangerous" and "driving to the right of the lane for no reason", and set the corresponding scene as the risk candidate scene. For example, based on the speech "the response to the cutting-in vehicle just now is slow and dangerous", the control unit 15 sets the scene of "the vehicle running in another lane cuts into the lane" at the latest time in the data traced back from the time when the speech was generated as the risk candidate scene. In addition, the data from the time when the vehicle starts to approach the lane to the time when the speech is generated may be automatically determined. Alternatively, based on the speech "driving to the right of the lane for no reason", the control unit 15 sets the scene of "the vehicle is approaching the right side of the lane" at the latest time in the data traced back from the time when the speech was generated as the risk candidate scene. Furthermore, data from the time when the vehicle 2 was traveling in the center of the lane before starting the behavior to the time when the utterance was made may be automatically specified.

[0037] The control unit 15 extracts the input value and the indication value of the machine learning model 120 in at least one risk candidate scenario. The control unit 15 refers to the storage unit 16 and extracts the input value and the indication value of the machine learning model 120 that are consistent with the time range in which the risk candidate scenario is generated as the extracted data D. The time range in which the risk candidate scenario is generated can also be set to a specified time before and after the timing of the generation of the risk candidate scenario.

[0038] The control unit 15 reports the monitoring information to the passengers of the vehicle 2 via the user interface 13. The monitoring information is information for monitoring the autonomous driving using the machine learning model 120, and includes input values ​​and indication values ​​of the machine learning model 120 extracted in the risk candidate scenario.

[0039] The control unit 15 causes the transmission unit 17 to transmit the extracted data D to the server 4. The transmission unit 17 is a communication device capable of wireless communication. Thus, only the extracted data is uploaded. The uploaded information can be used for early risk discovery and response.

[0040] Alternatively, the control unit 15 may cause the transmission unit 17 to transmit the extracted data D to the server 4 only with the permission of the passenger of the vehicle 2. For example, the monitoring information includes request information for selecting whether to permit the transmission to the server 4. The control unit 15 presents the request information together with the fact that the extracted data D has been extracted to the passenger of the vehicle 2. In response to the response of the permission to transmit to the server 4 being accepted via the user interface, the control unit 15 causes the transmission unit 17 to transmit the extracted data D to the server 4. In this way, the extracted data D may be uploaded only with the permission of the user. In this case, only the permitted information can be uploaded, thereby reducing communication resources and avoiding the pressure on data capacity.

[0041] Alternatively, the control unit 15 presents at least one risk candidate scenario to the passengers of the vehicle 2 via the user interface 13, and causes the sending unit 17 to send the extracted data D corresponding to the selected risk candidate scenario to the passengers of the vehicle 2 via the user interface 13. The control unit 15 presents the risk candidate scenarios generated during a specified period to the passengers of the vehicle 2. Alternatively, the control unit 15 presents the risk candidate scenarios that have been classified. For example, the control unit 15 calculates the similarity between the risk candidate scenarios based on the input values ​​and indication values ​​in the risk candidate scenarios, and classifies the risk candidate scenarios into categories based on the calculated similarities. Thus, the control unit 15 can present the risk candidate scenarios to the passengers of the vehicle 2 according to each of the categories into which they are classified.

[0042] The passenger of the vehicle 2 selects a risk candidate scene to be uploaded from the presented risk candidate scenes. The user interface 13 accepts the selection operation of the passenger. The control unit 15 generates the extraction data D corresponding to the selected risk candidate scene. Then, the control unit 15 causes the transmission unit 17 to transmit the extraction data D. In this case, the required information can be selected for uploading, so that the communication resources can be reduced and the pressure on the data capacity can be avoided.

[0043] Alternatively, the control unit 15 receives the ranking of at least one risk candidate scenario specified by the passenger of the vehicle 2 via the user interface 13, and determines the extraction data D to be sent by the sending unit 17 based on the ranking. For example, the passenger of the vehicle 2 pre-specifies the priority of the above-mentioned risk candidate scenarios as the ranking. When multiple risk candidate scenarios are determined, the control unit 15 determines the risk candidate scenario based on the ranking and generates the extraction data D. Thus, the control unit 15 can generate the extraction data D corresponding to the risk candidate scenario with a high ranking (high priority) when there is a limit on the communication volume.

[0044] Alternatively, the control unit 15 may change the size of the extracted data D according to the risk candidate scenario. For example, the control unit 15 may set a time range for extracting the extracted data D for each risk candidate scenario. Alternatively, in a situation where the automatically extracted extracted data D is sent in a fully permitted state, when an indication that a certain scene is a risk candidate scenario is received from a passenger of the vehicle 2 via the user interface 13, the control unit 15 may extract data in a manner that includes more information than the extracted data D corresponding to the risk candidate scenario for which the indication was not received.

[0045] Next, the operation of the automatic driving system 1 will be described. Figure 2 This is a flowchart showing an example of the operation of an autonomous driving system according to one embodiment. Figure 2 The flowchart shown is started when the automatic driving system 1 receives a start instruction operation during automatic driving performed by the automatic driving system 1 .

[0046] like Figure 2 As shown, first, as step S10, the control unit 15 of the automatic driving system 1 determines whether there is a risk candidate scene. The control unit 15 determines that there is a risk candidate scene when it detects driver intervention (accelerator, brake, etc.), sudden acceleration and deceleration, steering angle greater than or equal to a threshold, the distance between vehicle 2 and surrounding vehicles is less than or equal to a threshold, the distance between vehicle 2 and surrounding dividing lines is less than or equal to a threshold, the horn is sounded, the driver is frightened, etc. When it is determined that there is a risk candidate scene (step S10: yes), as step S12, the control unit 15 extracts the input and output information of the machine learning model 120. Thereby, the extracted data D is generated. Then, as step S14, the control unit 15 reports the monitoring information to the user. The control unit 15 displays the monitoring information including the fact that the extracted data D has been generated via the user interface 13, or reports it by voice. When step S14 ends, and when it is determined that there is no risk candidate scene (step S10: no), Figure 2 The flowchart shown ends. Figure 2The flowchart shown enables the user to appropriately monitor the autonomous driving achieved by the machine learning model 120.

[0047] Next, another operation of the automatic driving system 1 will be described. Figure 3 This is a flowchart showing an example of the operation of an autonomous driving system according to one embodiment. Figure 3 The flowchart shown is started when the automatic driving system 1 receives a start instruction operation during automatic driving performed by the automatic driving system 1 .

[0048] Figure 3 Steps S20, S22, S24 and Figure 2 The steps S10, S12, and S14 shown are the same.

[0049] When step S24 is completed, the control unit 15 determines, as step S26, whether there is user permission related to the transmission of the extracted data D. For example, the control unit 15 displays a Send Yes / No button as request information while displaying the monitoring information, and allows the user to select the Send Yes / No button. When it is determined that there is user permission (step S26: Yes), the control unit 15 causes the transmission unit 17 to operate to upload the extracted data D to the server 4 as step S28. When step S28 is completed, when it is determined that there is no risk candidate scenario (step S20: No), and when it is determined that there is no user permission (step S26: No), Figure 3 The flowchart shown ends. Figure 3 The flowchart shown can upload the extracted data D only with the user's permission.

[0050] Next, another operation of the automatic driving system 1 will be described. Figure 4 This is a flowchart showing an example of the operation of an autonomous driving system according to one embodiment. Figure 4 The flowchart shown is started when the automatic driving system 1 receives a start instruction operation during automatic driving performed by the automatic driving system 1 .

[0051] Figure 4 Steps S30, S32, S34 and Figure 2 The steps S10, S12, and S14 shown are the same. Figure 4 Step S36 shown is the same as Figure 3 The step S26 shown is the same.

[0052] If it is determined that the user has given permission (step S36: Yes), the control unit 15 determines whether there is a selection of the extracted data D as step S38. For example, the control unit 15 presents the risk candidate scenario to the user via the user interface 13. The user interface 13 accepts the user's selection operation of the risk candidate scenario. If it is determined that there is a selection of the risk candidate scenario (step S38: Yes), the control unit 15 extracts the data corresponding to the selected risk candidate scenario and generates the extracted data D as step S40. Then, the control unit 15 causes the sending unit 17 to operate to upload the extracted data D to the server 4.

[0053] When it is determined that no risk candidate scenario has been selected (step S38 : No), the control unit 15 extracts data corresponding to all presented risk candidate scenarios and generates extracted data D as step S42 . Then, the control unit 15 activates the transmission unit 17 to upload the extracted data D to the server 4 .

[0054] When steps S40 and S42 are completed, when it is determined that there is no risk candidate scenario (step S30: No), and when it is determined that there is no user permission (step S36: No), Figure 4 The flowchart shown ends. Figure 4 The flowchart shown can upload the extracted data D of the risk candidate scenario selected by the user.

[0055] [Summary of Implementation Methods]

[0056] According to the automatic driving system 1, at least one risk candidate scenario is determined based on at least one of the detection results of the external sensor 10, the detection results of the internal sensor 11, and the driving state of the occupants of the vehicle 2. Then, the input value and the indication value of the machine learning model 120 in at least one risk candidate scenario are extracted. Then, monitoring information containing the extracted input value and the indication value is reported to the occupants of the vehicle 2. In this way, the automatic driving system 1 can automatically determine the risk candidate scenario and report the monitoring information to the user (the occupant of the vehicle), thereby enabling the user to appropriately monitor the automatic driving achieved by the machine learning model 120.

[0057] As mentioned above, although the exemplary embodiment was described, it is not limited to the said exemplary embodiment, Various omissions, substitutions, and changes are possible.

[0058] For example, it is not necessary to obtain the passenger's permission for uploading at the timing of each monitoring information report, and the permission may be fully implemented when the automatic driving system 1 is activated.

[0059] The server 4 may also be configured as a server group. Furthermore, the server 4 may also include a data server that aggregates data uploaded from each vehicle. The data server may also collect information from each vehicle and publish the information via a web page or the like. Thus, the ranking of risk candidate scenarios taking into account the information of each vehicle may be posted on a web page. The user's permission or selection in the above-mentioned embodiment may also be performed via the posted web page. A mechanism may also be introduced in which a passenger who selects a scenario that many passengers have selected as a risk candidate scenario may receive rewards such as points.

Claims

1. An automatic driving system comprising: External sensors detect the vehicle’s external environment; An internal sensor for detecting the driving state of the vehicle; a machine learning model that takes the detection results of the external sensor and the detection results of the internal sensor as input values ​​and outputs an indication value of the autonomous driving of the vehicle; a driving unit that automatically drives the vehicle based on the indication value output by the machine learning model; a user interface for exchanging information with an occupant of said vehicle; as well as a control unit connected to the user interface, The control unit is configured as follows: determining at least one risk candidate scenario based on at least one of the detection result of the external sensor, the detection result of the internal sensor, and the driving state of an occupant of the vehicle; extracting the input value and the indicator value of the machine learning model in the at least one risk candidate scenario; as well as The user interface is caused to report monitoring information including the extracted input value and the instruction value to a passenger of the vehicle.

2. The automatic driving system according to claim 1, further comprising: a transmitting unit, transmitting the monitoring information to a device outside the vehicle, The monitoring information includes request information for selecting whether to permit transmission to the device outside the vehicle, The control unit is configured to, in response to receiving a response permitting transmission to the device outside the vehicle via the user interface, cause the transmission unit to transmit the extracted data to the device outside the vehicle, wherein: The extracted data is the extracted input value and the indication value of the machine learning model.

3. The automatic driving system according to claim 2, wherein: The control unit is configured to present the at least one risk candidate scenario to an occupant of the vehicle via the user interface, and cause the transmission unit to transmit the extracted data corresponding to the selected risk candidate scenario to the occupant of the vehicle via the user interface.

4. The automatic driving system according to claim 3, wherein: The control unit is configured to receive a ranking of the at least one risk candidate scenario specified by an occupant of the vehicle via the user interface, and determine the extracted data to be transmitted by the transmission unit based on the ranking.

5. The automatic driving system according to claim 3, wherein: The control unit is configured as follows: calculating the similarities between the risk candidate scenarios based on the input value and the indication value in the risk candidate scenarios; classifying the risk candidate scenarios into categories based on the calculated similarities; as well as The risk candidate scenarios are presented to an occupant of the vehicle via the user interface in each of the divided categories.

6. The automatic driving system according to claim 3, wherein: The control unit is configured to extract data such that extracted data corresponding to a risk candidate scene for which an indication is received from a passenger of the vehicle via the user interface that the scene is a risk candidate contains more information than extracted data corresponding to a risk candidate scene for which no indication is received.

Images