On-vehicle device, control method, and program

Abstract

To reduce a burden on a driver during automatic travel control.SOLUTION: An on-vehicle device comprises: a monitoring unit that recognizes objects around a vehicle and states of the objects on the basis of output from an external sensor mounted on the vehicle; a vehicle control unit that controls the vehicle according to the objects and their states; an output control unit that causes an output device to output control content and cause information indicating at least one of the objects and the states that have caused the control while the control is being executed; and a reception unit that receives an input for changing the control being executed, the input according to the cause information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle device, a control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a driving control device for controlling the driving of a vehicle. The driving control device is configured to reduce the discomfort and anxiety that drivers and passengers feel regarding automatic driving control. Specifically, the driving control device has a means for determining driving behavior details to be taken by the vehicle based on external recognition information received from external sensors and vehicle information including the vehicle's position and driving speed, a means for identifying driving behavior factors that contributed to the determination of the driving behavior details, and a means for outputting the driving behavior details and the driving behavior factors. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-199439 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 allows the driver of a vehicle to understand the details of control by the automatic driving control (driving behavior details) and the causes of the control (driving behavior factors). However, Patent Document 1 does not disclose any processing to be performed after the details are understood.

[0005] For example, there may be a problem with the cause of the control (such as a false detection), making that control unnecessary. Also, there may be no problem with the content or cause of the computer-generated control, but for some reason unrecognized by the computer, that control may become unnecessary. For example, the vehicle may slow down and stop in front of the crosswalk (control content) after detecting multiple pedestrians standing near the crosswalk ahead (cause), but the pedestrians may simply be having a conversation and have no intention of crossing the street. In this case, the driver can understand the situation through communication between the pedestrians and the driver. However, it is difficult for a computer to understand such a situation.

[0006] One example of how to handle such a situation is to switch from automated driving to manual driving. However, if a switch to manual driving is required every time such a situation occurs, the burden on the driver increases and the benefits of automated driving are diminished. Furthermore, if the vehicle is fully automated and the driver does not have the driving skills, the transition from automated driving to manual driving is not possible.

[0007] One example of an object of the present invention is to reduce the burden on the driver during automatic driving control. [Means for solving the problem]

[0008] The present invention provides a monitoring unit that monitors the state of the object based on the output of a sensor mounted on the vehicle; a generation unit that generates processing information for causing the vehicle to execute a process according to the state; a control unit that causes an output device to output cause information indicating at least one of the object and the state that caused the processing; a receiving unit that receives an input for changing the process caused by the cause indicated in the cause information; and The generating unit is an in-vehicle device that generates processing information in which the processing has been changed based on the reception result of the receiving unit.

[0009] The present invention provides The computer a monitoring step of monitoring the state of the object based on the output of a sensor mounted on the vehicle; a generating step of generating processing information for causing the vehicle to execute a process according to the state; a control step of causing an output device to output cause information indicating at least one of the object and the state that caused the processing; a receiving step of receiving an input for changing the process caused by the cause indicated in the cause information; Run In the generating step, processing information in which the processing has been changed is generated based on the reception result of the receiving step.

[0010] The present invention provides Computer, a monitoring means for monitoring the state of the object based on the output of a sensor mounted on the vehicle; a generating means for generating processing information for causing the vehicle to execute a process according to the state; a control means for causing an output device to output cause information indicating at least one of the object and the state that caused the processing; a receiving means for receiving an input for changing the process caused by the cause indicated in the cause information; It functions as The generating means is a program that generates processing information in which the processing has been changed based on the reception result of the receiving means. [Brief explanation of the drawings]

[0011] The above-mentioned objects, as well as other objects, features and advantages, will become more apparent from the preferred embodiments described below and the accompanying drawings.

[0012] [Figure 1] FIG. 2 is a diagram showing an example of a functional block diagram of the car window display device of the present embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an in-vehicle device according to an embodiment of the present invention; [Figure 3] FIG. 2 is a diagram illustrating an example of data processed by the in-vehicle device according to the present embodiment. [Figure 4] FIG. 2 is a diagram schematically illustrating an example of an image output by the in-vehicle device of the present embodiment. [Figure 5] FIG. 2 is a diagram schematically illustrating an example of an image output by the in-vehicle device of the present embodiment. [Figure 6] FIG. 2 is a diagram schematically illustrating an example of an image output by the in-vehicle device of the present embodiment. [Figure 7] 4 is a flowchart showing an example of a processing flow of the in-vehicle device of the present embodiment. [Figure 8] FIG. 2 is a diagram schematically illustrating an example of an image output by the in-vehicle device of the present embodiment. [Figure 9] FIG. 2 is a diagram schematically illustrating an example of an image output by the in-vehicle device of the present embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of data processed by the in-vehicle device according to the present embodiment. [Figure 11] 4 is a flowchart showing an example of a processing flow of the in-vehicle device of the present embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of data processed by the in-vehicle device according to the present embodiment. [Figure 13] 4 is a flowchart showing an example of a processing flow of the in-vehicle device of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, like components are designated by like reference numerals, and the description thereof will be omitted as appropriate.

[0014] First, an outline of this embodiment will be described. The on-board device of this embodiment monitors the state of an object (e.g., a pedestrian, a preceding vehicle, etc.) based on the output of a sensor mounted on the host vehicle. Then, the on-board device of this embodiment causes the host vehicle to execute processing (e.g., deceleration, stopping, backing up, changing lanes, accelerating, changing course) according to the state of the object.

[0015] The in-vehicle device of this embodiment also causes the output device to output cause information indicating at least one of the object and state that caused the processing. Furthermore, the in-vehicle device of this embodiment accepts inputs to change the processing caused by the cause indicated in the cause information. For example, the in-vehicle device accepts inputs to cancel the processing caused by the cause, or inputs to change the recognition results of the object or state that caused the processing. The in-vehicle device of this embodiment then controls the vehicle in accordance with the inputs.

[0016] To further understand the outline of this embodiment, an example will be described. For example, when the in-vehicle device of this embodiment detects that one or more pedestrians are standing near a crosswalk ahead, it decelerates and stops the vehicle in front of the crosswalk accordingly. Then, the in-vehicle device of this embodiment notifies the driver of the cause of the "deceleration and stop" that is currently being performed, such as "the pedestrians," "the fact that the pedestrians are standing near the crosswalk," or "the pedestrian is standing near the crosswalk." This notification allows the driver to understand the cause of the "deceleration and stop" that the vehicle is currently performing.

[0017] Here, let us assume that, through subsequent communication between the pedestrians and the driver, the driver realizes that the pedestrians are just having a conversation and have no intention of crossing the street.

[0018] In this case, the driver can make an input to change the processing (deceleration / stop) caused by the notified cause. For example, the driver can make an input to cancel the processing caused by the cause or to change the recognition result related to the cause (e.g., pedestrian or its state). The in-vehicle device then executes processing according to the input. For example, if the cause of the processing (deceleration / stop) caused by the cause is eliminated by canceling the processing (deceleration / stop) caused by the cause or changing the recognition result, the in-vehicle device cancels the execution of the processing (deceleration / stop). The in-vehicle device then controls the vehicle based on the state after cancellation. For example, the in-vehicle device starts or accelerates the vehicle.

[0019] In this way, according to the in-vehicle device of this embodiment, in the above-mentioned situation, the vehicle can continue to operate under automatic driving control simply by making a predetermined input without switching to manual driving, thereby reducing the burden on the driver.

[0020] Next, the configuration of the in-vehicle device of this embodiment will be described in detail.

[0021] An in-vehicle device is a device that is mounted on a vehicle and controls the vehicle itself. An example of an in-vehicle device is an ECU (Electronic Control Unit). Note that the term "vehicle" used in the following description refers to a vehicle that is the target of control by the in-vehicle device.

[0022] 1 shows an example of a functional block diagram of an in-vehicle device 10 according to this embodiment. As shown in the figure, the in-vehicle device 10 includes a monitoring unit 11, a generating unit 12, a control unit 13, a receiving unit 14, and an output unit 15.

[0023] First, an example of the hardware configuration of the in-vehicle device 10 that realizes these functional units will be described. Each functional unit is realized by any combination of hardware and software, centered around a CPU (Central Processing Unit) of any computer, memory, programs loaded into the memory, a storage unit such as a hard disk that stores the programs (this can store programs that are pre-loaded when the device is shipped, as well as programs downloaded from storage media such as CDs (Compact Discs) or servers on the Internet), and a network connection interface. Those skilled in the art will understand that there are many variations in the realization methods and devices.

[0024] 2 is a block diagram illustrating the hardware configuration of the in-vehicle device 10 of this embodiment. As shown in FIG. 2, the in-vehicle device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. Note that the in-vehicle device 10 does not necessarily have to have the peripheral circuit 4A.

[0025] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to transmit and receive data to and from each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input / output interface 3A includes interfaces for acquiring information from input devices (e.g., keyboard, mouse, microphone, etc.), external devices, external servers, external sensors, etc., and interfaces for outputting information to output devices (e.g., display, speaker, printer, mailer, etc.), external devices, external servers, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0026] Next, the function of each functional unit shown in FIG. 1 will be described in detail.

[0027] The monitoring unit 11 monitors the state of an object based on the output of a sensor mounted on the vehicle. The monitoring unit 11 acquires output from a sensor that collects information about the external environment of the vehicle, such as a camera (e.g., a camera that captures the outside of the vehicle), a LIDAR (laser radar), or a radar. Alternatively, the monitoring unit 11 may acquire information collected by a sensor installed on the road through road-to-vehicle communication.

[0028] The monitoring unit 11 then analyzes the sensor output and recognizes the object and its state. For example, the monitoring unit 11 may extract the object from the image captured by the camera using the feature amounts of the appearance of each of a plurality of objects stored in advance. The object is an object that affects the control of the vehicle, and examples thereof include, but are not limited to, pedestrians, other vehicles, vehicles ahead, oncoming vehicles, bicycles, fallen objects, traffic lights, road signs, crosswalks, landmarks, etc.

[0029] Then, the monitoring unit 11 recognizes the state of each extracted object. For example, multiple states may be prepared for each object in advance, and the monitoring unit 11 may determine which state the extracted object is in. The state of the object can be determined based on the situation around the object, the situation of the object itself, etc.

[0030] When the object is a pedestrian, the state may be, for example, "planning to cross the crosswalk ahead," "currently crossing the crosswalk ahead," "already crossed the crosswalk ahead," "walking on the sidewalk," "other," etc., but is not limited to these. The monitoring unit 11 can determine the state of the pedestrian based on, for example, the situation around the pedestrian (e.g., whether a crosswalk exists or not), the situation of the pedestrian itself (e.g., whether or not the pedestrian is facing the crosswalk, whether or not the pedestrian is moving toward the crosswalk, where the pedestrian is walking, etc.), etc.

[0031] Furthermore, when the target object is a vehicle ahead, examples of the state include, but are not limited to, "driving," "temporarily stopped," "parked," "decelerating," "other," etc. The monitoring unit 11 can determine the state of the target vehicle based on, for example, the situation around the target vehicle (e.g., the traffic light ahead is red, a pedestrian is crossing ahead), the situation of the target vehicle itself (e.g., whether the vehicle is stopped, whether the brake lights are on, whether the engine is running), etc.

[0032] The monitoring unit 11 can also identify the position within the image (position within the frame) for each extracted object. Furthermore, the monitoring unit 11 can identify the relative position of each extracted object with respect to the vehicle (position of the lidar or radar) based on the output of the lidar, radar, etc.

[0033] The monitoring unit 11 can register the results of the above-described processing. Examples of the registered contents are shown in Fig. 3 and Fig. 10. In the example shown in Fig. 3, the type of the extracted object, its position, its state, and whether or not a release input has been made are associated with each other. In the example shown in Fig. 10, the type of the extracted object, its position, and its state are associated with each other.

[0034] The object column records the type of each of the multiple extracted objects. For example, pedestrians, vehicles ahead, bicycles, etc. are recorded. The position column records the position of each of the multiple extracted objects, such as a position in an image captured by a camera, or a relative position based on the vehicle itself detected by a lidar, radar, etc. The status column records the status recognized in the above process. For example, "planning to cross the crosswalk ahead," "currently crossing the crosswalk ahead," etc. are recorded. The cancellation input presence / absence column records information indicating whether the reception unit 14, which will be described below, has received a cancellation input.

[0035] Here, an example of the process in which the monitoring unit 11 registers and updates the registration information in Fig. 3 and Fig. 10 will be described with reference to the flowchart in Fig. 7. Note that the process described is merely an example, and the present invention is not limited to this.

[0036] First, the monitoring unit 11 acquires an image (sensor information) of a frame to be processed from a video captured by a camera (S10).

[0037] The monitoring unit 11 then recognizes objects appearing in the image of the frame to be processed (image captured by the camera) (S11). After that, the monitoring unit 11 determines whether each recognized object is the same object as an object recognized in the image of the previous frame (e.g., an object registered in the registration information at that time (e.g., the information in Figures 3 and 10)). This determination can be made using any conventional technology.

[0038] If the object is not the same as the object recognized in the image of the previous frame, the monitoring unit 11 issues new identification information (e.g., a serial number) and associates the identification information with the object recognized in the image of the frame to be processed. On the other hand, if the object is the same as the object recognized in the image of the previous frame, the monitoring unit 11 associates the object recognized in the image of the frame to be processed with the identification information of the object that has already been issued.

[0039] Then, the monitoring unit 11 associates the identification information of the object extracted from the image of the frame to be processed with the position of the object in the image.

[0040] Furthermore, the monitoring unit 11 determines the state of each object extracted from the image of the frame to be processed (S12). Then, the monitoring unit 11 associates the state recognition result with the identification information of the object recognized in the image of the frame to be processed.

[0041] As described above, the monitoring unit 11 updates the registration information (for example, the information in FIGS. 3 and 10) based on the recognition result obtained from the image of the frame to be processed (S13).

[0042] If the object recognized in the image of the frame to be processed is not the same object as the object recognized in the image of the previous frame, the monitoring unit 11 newly registers the identification information of the object recognized in the image of the frame to be processed, its position in the image, and its state in the registration information (e.g., information in Figures 3 and 10).

[0043] On the other hand, if the object recognized in the image of the frame to be processed is the same object as the object recognized in the image of the previous frame, the monitoring unit 11 updates the information of the object registered in the registration information (e.g., information in Figures 3 and 10) based on the identification information of the object recognized in the image of the frame to be processed, its position in the image, and the status determination result.

[0044] Furthermore, the monitoring unit 11 can delete information that satisfies a predetermined condition from the registered information (e.g., the information in Figs. 3 and 10). For example, the monitoring unit 11 may delete, from the registered information (e.g., the information in Figs. 3 and 10), information relating to an object that does not match an object recognized in the image of the frame to be processed, that is, information relating to an object that was not recognized in the image of the frame to be processed.

[0045] The monitoring unit 11 repeats the above-mentioned processing, for example.

[0046] Note that the above processing is merely an example, and other processing may be adopted as long as it can achieve similar results. For example, in the processing of S11, an object tracking function that is widely known in image processing may be used to recognize an object that appears in the image of the previous frame in the image of the frame to be processed. Also, a new object that has not been extracted in the previous frame may be recognized in the image of the frame to be processed by using the appearance feature of a pre-registered object.

[0047] Returning to Figure 1, while the vehicle is performing a predetermined process (e.g., decelerating, stopping, backing up, changing lanes), the control unit 13 causes the output device to output cause information indicating at least one of the object and state that caused the process.

[0048] The control unit 13 can identify the object and state that caused the processing based on the registration information (e.g., the information in FIGS. 3 and 10), and output cause information indicating the details thereof. For example, in the case of the registration information shown in FIG. 3, the control unit 13 identifies an object that is indicated in the status column as being in a state that causes the host vehicle to perform a predetermined processing (e.g., decelerate, stop, reverse, change lanes) and that is indicated as not having received a cancel input in the cancellation input presence / absence column. Then, the control unit 13 identifies the identified object and the state of the object as the cause of the processing. On the other hand, in the case of the registration information shown in FIG. 10, the control unit 13 identifies an object that is indicated in the status column as being in a state that causes the host vehicle to perform a predetermined processing (e.g., decelerate, stop, reverse, change lanes). Then, the control unit 13 identifies the identified object and the state of the object as the cause of the processing.

[0049] Note that each of the multiple states of the object may be associated in advance with the process to be executed when each state is detected. Based on such correspondence information, the control unit 13 may, for example, identify the object and state that caused each process from the information registered in the registration information (e.g., the information in FIGS. 3 and 10).

[0050] For example, processes such as "slow down" and "stop" may be defined in response to "(object) pedestrian: (status) planning to cross the crosswalk ahead" and "(object) pedestrian: (status) currently crossing the crosswalk ahead."

[0051] Furthermore, processes such as "slow down," "stop," "change lanes," and "follow the vehicle ahead" may be defined in response to "(object) vehicle ahead: (state) temporarily stopped." Furthermore, processes such as "slow down," "stop," "change lanes," and "back up" may be defined in response to "(object) obstacle obstructing driving: (state) stopped."

[0052] Examples of the output device include, but are not limited to, a display device installed in a vehicle, a head-up display device, a head-mounted display device, a projection device, a smartphone, a tablet, a speaker, etc. The in-vehicle device 10 may have the output device. Alternatively, the output device may be configured to be physically and / or logically separated from the in-vehicle device 10. When the output device and the in-vehicle device 10 are configured to be physically and / or logically separated, the output device and the in-vehicle device 10 are configured to be able to communicate with each other via wire and / or wirelessly.

[0053] FIG. 4 shows an example of output by the control unit 13. In the example shown, the control unit 13 outputs an image in which cause information is superimposed on a real-time image captured by the camera. In the figure, "Reason for stopping 1" and "Reason for stopping 2" shown in association with the two pedestrians located on the left are the cause information. The display position of the cause information within the image is determined based on the positions of the two pedestrians (objects) within the image (see FIG. 3).

[0054] From the image shown in FIG. 4, the driver can see that the vehicle has slowed down and stopped due to the presence of two pedestrians near the crosswalk.

[0055] FIG. 5 shows another example of output by the control unit 13. In the example shown, the control unit 13 outputs an image in which cause information is superimposed on a real-time image captured by a camera. In the figure, "Stopping reason 1" shown in association with a vehicle located in front of the vehicle is the cause information. Note that the vehicle in front is stopped at the side of the road. The display position of the cause information in the image is determined based on the position of the vehicle in front (object) in the image (see FIGS. 3 and 10).

[0056] According to the image shown in FIG. 5, the driver can see that his vehicle has slowed down and stopped due to the presence of a stopped vehicle ahead that is located in the same lane as his vehicle.

[0057] As another output example, as shown in FIG. 6, the control unit 13 may cause the output device to output text information such as "A stopped vehicle ahead has been detected, so the vehicle will decelerate and stop." The sentence may be output through a speaker. In this case, as shown in FIG. 8, a display may be performed that associates the text information with the object that caused the problem. Furthermore, although not shown, if there are multiple causes of the processing, the output device may output text information corresponding to each of the multiple causes.

[0058] The control unit 13 may output the above-described information before the vehicle executes the process due to the cause. In this case, as shown in FIG. 9, text information may be output to notify the driver of the process, such as "A stopped vehicle ahead has been detected, so the vehicle will decelerate and stop in ○ seconds." This allows the driver to prepare mentally. Furthermore, by accepting an input to change the process due to the cause based on such advance notice information, it is possible to avoid the execution of unnecessary processes (e.g., stop, stop, etc.). In this case, as shown in FIG. 8, a display may be displayed that associates the text information with the object that caused the error.

[0059] Furthermore, when a head-up display device is used, the cause information as described above may be displayed at a predetermined position on the windshield of the vehicle. That is, cause information corresponding to each object (real object) can be displayed at a position on the windshield corresponding to each object (real object) visible through the windshield from the driver's perspective (for example, the intersection of the straight line connecting the driver's eye position and the position of the object (real object) with the windshield). Means for realizing such a display can be realized based on conventional technology.

[0060] 1, the reception unit 14 receives an input to change a process caused by a cause indicated in the cause information output by the control unit 13. For example, the reception unit 14 receives an input to cancel a process caused by the cause, or an input to change the object that caused the cause or its state. The reception unit 14 can receive the input via any input device, such as a touch panel display device, operation buttons, a camera, a microphone, or a gaze detection device.

[0061] For example, images such as those shown in Figures 4 to 6, 8, and 9 may be output via a touch panel display device. The reception unit 14 may then receive an input via the touch panel display device, such as touching characters such as "Stop Reason 1" and "Stop Reason 2" shown in Figures 4 and 5, or an object corresponding to the characters. The reception unit 14 may then receive the input as an input to cancel the execution of a process caused by the cause of the touch. In response to the input, for example, the registration information shown in Figure 3 is updated. That is, the fact that the cancellation has been made is registered in the "Cancel Input Presence / Absence" field in accordance with the cause of the touch (a predetermined state of a predetermined object).

[0062] Additionally, in response to an input of touching characters such as "Stop Reason 1" or "Stop Reason 2" or an object corresponding to the characters, the control unit 13 may cause the output device to output guidance such as "Do you want to cancel the execution of the process (deceleration, stop) caused by the cause? Yes or No." The reception unit 14 may then accept an input of "Yes" in response to the guidance as an input to cancel the execution of the process caused by the cause. In response to the input, for example, the registration information shown in FIG. 3 is updated. That is, the fact that cancellation has been made is registered in the "Cancel Input Presence / Absence" field in accordance with the touched cause (a predetermined state of a predetermined object).

[0063] Additionally, in response to input of touching characters such as "Stop reason 1" or "Stop reason 2" or an object corresponding to the characters, control unit 13 may cause the output device to output guidance such as "Do you want to change the recognition result? Yes or NO." Then, when reception unit 14 receives the input of "Yes," control unit 13 may cause the output device to output information for changing the recognition result.

[0064] For example, the control unit 13 may output the current recognition result. The recognition result may include a recognition result of an object and a recognition result of a state. Specifically, examples include, but are not limited to, "pedestrians who plan to cross," "pedestrians currently crossing," and "vehicles temporarily stopped."

[0065] Furthermore, the control unit 13 may output a list of candidates for the recognition result after the change. For example, if the current recognition result is "pedestrian planning to cross" or "pedestrian crossing", "waiting pedestrian", "traffic control person", etc. may be output as a list of candidates for the recognition result after the change. Furthermore, if the current recognition result is "temporarily stopped vehicle", "parked vehicle", "broken down vehicle", etc. may be output as a list of candidates for the recognition result after the change.

[0066] Then, the accepting unit 14 may accept the changed recognition result from the list of candidates that has been output. In response to the input, for example, the registered information shown in Fig. 3 or 10 is updated. That is, the information in the object column, the information in the status column, etc. are updated in response to the cause of the touch (a predetermined status of a predetermined object).

[0067] As another example, the control unit 13 may output the current recognition result and cause the output device to output guidance such as "Which recognition result would you like to change? Object or Status." Then, when the reception unit 14 receives input of "object," it may output a list of candidate objects after the change. For example, if the current recognition result is "pedestrian," a "statue," a "doll," a "traffic controller," etc. may be output as the list of candidate objects after the change. Furthermore, if the current recognition result is "obstacle that obstructs driving," an "obstacle that is safe to step on" etc. may be output as the list of candidate objects after the change. On the other hand, when the reception unit 14 receives input of "status," it may output a list of candidate objects after the change. For example, if the current recognition result is "planned to cross," "waiting" etc. may be output as the list of candidate objects after the change.

[0068] Then, the accepting unit 14 may accept the changed recognition result from the list of candidates that has been output. In response to the input, for example, the registered information shown in Fig. 3 or 10 is updated. That is, the information in the object column, the information in the status column, etc. are updated in response to the cause of the touch (a predetermined status of a predetermined object).

[0069] Furthermore, in the case of the image shown in FIG. 6, the reception unit 14 may receive, via the touch panel display device, a message saying, "A stopped vehicle ahead has been detected, so decelerate and stop." or an input of touching an area corresponding to the message (e.g., an area where a square surrounding the message is displayed). The reception unit 14 may then receive the input as an input to cancel the execution of a process caused by the cause of the touch. Alternatively, the reception unit 14 may output, to the output device, a message such as, "Do you want to cancel the execution of the process (deceleration, stop) caused by the cause? Yes or NO," in response to the input. The reception unit 14 may then receive a "Yes" response to the message as an input to cancel the execution of a process caused by the cause. Alternatively, the reception unit 14 may receive an input to change the recognition result in the same manner as described above.

[0070] Alternatively, the receiving unit 14 may receive input similar to the example using the touch panel display device described above by receiving input to select a predetermined area on the image using an operation button and a cursor displayed on the image.

[0071] In addition, when the cause information is displayed on the windshield using a head-up display device, the reception unit 14 may receive the input of the above change by detecting a predetermined action of the driver using a camera (e.g., an action of touching the cause information displayed on the windshield, etc.).

[0072] Alternatively, the reason for stopping may be expressed by displaying an abstract map on the touch panel display device and arranging icons of the vehicle's position and detected objects (e.g., people, other vehicles, etc.).The acceptance unit 14 may then accept an input to change the processing caused by the cause by touching the icon, in the same manner as described above.

[0073] Alternatively, the reception unit 14 may use a microphone to acquire the voice of the passenger, analyze the acquired voice to detect a predetermined voice (i.e., identify the content of the speech), and accept the input of the change. For example, voice such as "release deceleration" or "pedestrian has no intention of crossing" may be detected.

[0074] Alternatively, the reception unit 14 may select an object for which the recognition result is to be changed by detecting the gaze of the passenger using a gaze detection device. In this case, the object in front of the gaze may be selected as the object for which the recognition result is to be changed. Note that the selection for the change may be made using another device, or may use the blink detection result of the gaze detection device.

[0075] 4 to 6, 8, and 9, the control unit 13 can cause the output device to output one or more pieces of cause information that caused the execution of a process (e.g., deceleration, stop). If there are multiple causes for the execution of a process (e.g., deceleration, stop), the control unit 13 can cause the output device to output all of the cause information.

[0076] The receiving unit 14 can then receive an input to change the processing (e.g., deceleration, stop) caused by each of one or more pieces of cause information. That is, the receiving unit 14 can individually receive an input to change the processing (e.g., deceleration, stop) caused by each of the multiple causes.

[0077] 1, the generation unit 12 generates processing information for causing the host vehicle to execute processing according to the state of the object. The generation unit 12 can also generate processing information in which the processing has been changed based on the reception result of the reception unit 14. The output unit 15 outputs the processing information generated by the generation unit 12 to a vehicle control device that controls the vehicle.

[0078] The generation unit 12 determines the control content of the host vehicle based on the registration information (e.g., the information in FIGS. 3 and 10) and various other information. Then, the generation unit 12 generates processing information for controlling the host vehicle based on the determined content. For example, elements of the host vehicle such as steering, braking, and acceleration are controlled according to the processing information.

[0079] Examples of the above-mentioned various other information include, but are not limited to, information indicating the position of the vehicle, map information, route information indicating the route to reach the destination, external information indicating the conditions outside and around the vehicle detected based on cameras, lidars, radars, etc., and sensor information (e.g., speed, etc.) from various sensors installed in the vehicle.

[0080] For example, if at least one object in a state that causes the vehicle to execute a predetermined process is registered in the registration information (e.g., information in Figures 3 and 10), the generation unit 12 may decide to cause the vehicle to execute that process.

[0081] An "object in a state that causes the host vehicle to execute a predetermined process" is, for example, an object that is indicated in the state column of the registration information in Fig. 3 as being in a state that causes the host vehicle to execute a predetermined process (e.g., decelerate, stop, reverse, change lanes), and is indicated in the cancellation input presence / absence column as not having received a cancellation input.Other examples of such an object are objects that are indicated in the state column of the registration information in Fig. 10 as being in a state that causes the host vehicle to execute a predetermined process (e.g., decelerate, stop, reverse, change lanes).

[0082] As described above, each of the multiple states of the object may be associated in advance with a process to be executed when each state is detected. Then, based on such correspondence information, the generation unit 12 may identify a cause (state of the object) for executing a predetermined process (e.g., deceleration, stop) from information registered in the registered information (e.g., information in FIGS. 3 and 10).

[0083] For example, when at least one object that is in a state that causes the host vehicle to decelerate or stop is registered, the generation unit 12 generates processing information for decelerating or stopping the host vehicle. Then, when the receiving unit 14 accepts input and the cause of the host vehicle decelerating or stopping disappears, the generation unit 12 stops the execution of deceleration or stopping of the host vehicle. In response to this, for example, the generation unit 12 generates processing information for starting or accelerating the host vehicle.

[0084] As described above, multiple processes may be defined for one pair of an object and a state, such as "slow down," "stop," "change lanes," and "follow the vehicle ahead" for "(object) vehicle ahead: (state) temporarily stopped." In this case, the control unit 13 may determine which process to execute based on, for example, the state of the host vehicle (e.g., moving, stopped) and the state around the host vehicle (e.g., whether there is a lane to change to, whether other lanes are in a state where a lane change is possible), etc.

[0085] Even if the receiving unit 14 receives input, if other causes for the deceleration or stopping of the vehicle (due to a specified state of another object) remain, the generating unit 12 continues to decelerate or stop the vehicle.

[0086] A modification of this embodiment will now be described.

[0087] After the receiving unit 14 receives an input to cancel the execution of a predetermined process (first process) caused by the state of the object (first object), the monitoring unit 11 may monitor the object (first object) using subsequent sensor output and detect a predetermined action by the object (first object). Then, in response to the detection of the predetermined action, the monitoring unit 11 cancels the "cancellation of the execution of the predetermined process (first process)." For example, in response to the detection, the monitoring unit 11 changes the content of the cancellation input presence / absence column of the registration information (e.g., FIG. 3) corresponding to the object (first object) to content indicating that the cancellation input has not been received.

[0088] Furthermore, after the receiving unit 14 receives an input to change the recognition result, the monitoring unit 11 may monitor the object (first object) using the subsequent sensor output and detect a predetermined action by the object (first object). Then, in response to the detection of the predetermined action, the monitoring unit 11 may change the recognition result, which has been changed by the receiving unit 14, to a new recognition result recognized by the monitoring unit 11. In response to this, the contents (e.g., object column, status column) of the registration information (e.g., information in FIGS. 3 and 10) may be updated.

[0089] The predetermined action to be detected may be determined in advance for each object or for each state of the object. For example, if the object is a pedestrian, movement may be the predetermined action. If the object is a vehicle, starting, turning on a blinker, turning off a hazard lamp, etc. may be the predetermined action. If the object is a pedestrian standing near a crosswalk, movement toward the crosswalk may be the predetermined action. Note that the examples given here are merely examples and are not limiting.

[0090] The generation unit 12 can control the vehicle based on the updated registration information (e.g., the information in FIGS. 3 and 10). In the above example in which the execution of the first process caused by the state of the first object is canceled (first cancellation) and then the first cancellation is canceled, the generation unit 12 processes the state of the first object as the cause of the first process before the first cancellation, does not process the state of the first object as the cause of the first process after the first cancellation, and again processes the state of the first object as the cause of the first process after the first cancellation is canceled.

[0091] Here, other specific examples will be described. Note that these specific examples are merely examples and are not intended to limit the scope of the present invention.

[0092] <Example 1> For example, suppose that while driving on a narrow alley, the monitoring unit 11 detects a parked vehicle ahead. Then, the generation unit 12 decides to reverse and take another road based on the detection result and the state of the surroundings of the vehicle (e.g., the width of the vehicle ahead, the road width, the available width, etc.) (i.e., it is determined that it is not possible to move forward while avoiding the vehicle ahead). In such a case, the control unit 13 outputs information indicating that the vehicle will reverse and take another road due to the presence of a parked vehicle ahead.

[0093] Here, suppose that the driver of the host vehicle confirms that the driver of the parked vehicle ahead has returned and determines that the vehicle will soon be able to move forward. The driver of the host vehicle then performs an input to change the process (backing up) caused by the displayed cause (the parked vehicle ahead). For example, the driver performs an input to change the recognition result of the state of the vehicle ahead from "parked" to "temporarily stopped." The generation unit 12 then re-determines the process to be executed by the host vehicle based on the changed content. For example, the process may be changed from "backing up" to "following the vehicle ahead" in response to the change in the recognition result from "parked" to "temporarily stopped."

[0094] <Example 2> For example, suppose that while driving, the monitoring unit 11 detects a stopped obstacle ahead that will obstruct driving. Then, the generation unit 12 decides to change lanes based on the detection result and the state around the vehicle (e.g., whether there is a lane to change to, whether other lanes are in a state where a lane change is possible), etc. In such a case, the control unit 13 outputs information indicating that the lane change will occur before the lane change. That is, due to the presence of an obstacle ahead that will obstruct driving, the control unit 13 outputs a notice that a lane change will occur in ○ seconds.

[0095] Here, it is assumed that the driver of the host vehicle determines that the obstacle ahead that is determined to obstruct driving is an obstacle that is safe to run over. Then, the driver of the host vehicle performs an input to change the process (lane change) caused by the displayed cause (a stopped obstacle ahead that obstructs driving). For example, the driver performs an input to change the recognition result from "obstacle obstructing driving" to "obstacle that is safe to run over." Then, the generation unit 12 re-determines the process to be executed by the host vehicle based on the changed content. For example, the "lane change" may be canceled as the recognition result of the object is changed from "obstacle obstructing driving" to "obstacle that is safe to run over."

[0096] <Example 3> For example, suppose that while traveling on a road with two or more lanes in each direction, the monitoring unit 11 detects a vehicle ahead traveling at a low speed. Then, the generation unit 12 decides to change lanes based on the detection result and the state around the vehicle (e.g., whether there is a lane to change to, whether other lanes are in a state where a lane change is possible), etc. In such a case, the control unit 13 outputs information indicating that the lane change will occur before the lane change. In other words, due to the presence of a vehicle ahead traveling at a low speed, a notice is output that a lane change will occur in ○ seconds.

[0097] Here, assume that the driver of the host vehicle determines that there is no particular effect in changing lanes due to traffic congestion. Then, the driver of the host vehicle performs an input to change the process (lane change) caused by the displayed cause (vehicle ahead traveling at a low speed). For example, the driver performs an input to cancel the lane change caused by that cause. Then, the generation unit 12 re-determines the process to be executed by the host vehicle based on the input content. For example, the "lane change" may be canceled, and another process such as "following the vehicle ahead" may be determined.

[0098] According to the in-vehicle device 10 of the present embodiment described above, the cause of the processing being executed by the vehicle can be output, thereby notifying the driver of the cause of the processing. Then, an input for changing the processing caused by the notified cause can be received from the user, and the automatic driving control can be continued based on the received input.

[0099] According to the in-vehicle device of this embodiment, even when there is a problem with the cause of the control (such as a false detection) and the control of that content is unnecessary, or when there is no problem with the content and cause of the computer-controlled control but the control based on that cause becomes unnecessary for some reason that the computer cannot recognize, the vehicle can continue to operate under automatic driving control by simply making a predetermined input without switching to manual driving. As a result, the burden on the driver during automatic driving control can be reduced.

[0100] Furthermore, according to the in-vehicle device of this embodiment, when there are multiple control causes, it is possible to individually accept inputs for changing the processing based on those causes. If inputs for changing the processing based on each of the multiple causes could be accepted all at once, human error could occur, such as overlooking a cause that should not be changed and is buried among the multiple causes. As a result, inconveniences such as causing an accident could occur. In the present embodiment, when there are multiple control causes, it is possible to individually accept inputs for changing the processing based on those causes, thereby alleviating such inconveniences.

[0101] Furthermore, according to the in-vehicle device 10 of this embodiment, even after receiving the input of the change, it is possible to continue monitoring the object related to the cause. Then, when the object performs a predetermined action, it is possible to cancel the cancellation based on the user input (cancellation of the execution of a predetermined process due to a predetermined cause), or to perform a process based on a new recognition result by a computer, for example.

[0102] In such a case, even if the driver mistakenly inputs a change to a predetermined process (e.g., deceleration, stop) due to a miscommunication between the driver and another person (e.g., a pedestrian, the driver of another vehicle, etc.), the system can cause the vehicle to re-execute the canceled process (e.g., deceleration, stop) in response to the object performing a predetermined action. As a result, accidents caused by human error by the driver can be reduced. In other words, a safe system can be realized.

[0103] Furthermore, the monitoring unit 11 may monitor the state of the object using the output of the sensor, and when a predetermined movement of the object is detected, may stop accepting an input to cancel the execution of a predetermined process caused by the state of the object. The flow of this process will be described with reference to FIG. 11.

[0104] Specifically, the monitoring unit 11 monitors the state of the object using the output of the sensor (S20). Then, the determination unit determines the monitoring accuracy (S21). Note that although the determination unit is not shown in FIG. 1, the in-vehicle device 10 may have a determination unit.

[0105] The monitoring accuracy in S21 is higher when the state of the object is clear and the processing that the vehicle should take is clear, for example, "a pedestrian is walking on the crosswalk," "a pedestrian does not reliably cross the crosswalk," or "a vehicle parked ahead does not depart (this can be determined by detecting hazard lights, blinkers, etc. from a captured image of the parked vehicle)." The states of objects classified as "those with high monitoring accuracy" may be registered in advance. Then, the determination unit may determine whether the state of the object recognized by the monitoring unit 11 is "those with high monitoring accuracy" based on the registered contents.

[0106] If the monitoring accuracy is high (Yes in S22), it is determined that there is no need to change the execution of the processing of the vehicle, and the reception unit 14 stops receiving the data (S23). If the monitoring accuracy is not high (No in S22), it is determined that the execution of the processing of the vehicle can be changed, and the reception unit 14 continues receiving the data (S24). As a result, accidents caused by driver error can be reduced in situations where it is necessary to stop or slow down the vehicle reliably. Furthermore, it is possible to reduce the processing load within the device in situations where it is not necessary to stop or slow down the vehicle reliably.

[0107] Alternatively, instead of the monitoring accuracy, the determination unit may determine the degree of danger of the monitoring results of the state of the object based on registered information (see FIG. 12) in which a degree of danger is preset for each state of the object. The degree of danger may be expressed numerically, for example, as a degree of danger from 1 to 5, or may be expressed in other ways, such as high, medium, low, etc. In this case, if the determination unit determines that the state of the object is at a predetermined level or higher, it is determined that changing the execution of the processing of the subject vehicle is dangerous, and the reception unit 14 stops receiving the request.

[0108] An example of the processing flow of the in-vehicle device 10 in this example will now be described using the flowchart in Fig. 13. The monitoring unit 11 monitors the state of the object using the output of the sensor (S30). Then, the determination unit determines the degree of danger of the state of the object based on registered information in which a degree of danger is set in advance for each state of the object (S32).

[0109] For example, the higher the risk level is set in the registered information for an object state that is more likely to pose a danger to the driver of the vehicle or the object when the execution of the process is canceled, such as "a pedestrian is walking on a crosswalk." On the other hand, the lower the risk level is set in the registered information for an object state that is less likely to pose a danger to the driver of the vehicle or the object when the execution of the process is canceled, such as "a pedestrian may cross the crosswalk." The determination unit may determine whether the state of the object recognized by the monitoring unit 11 is "high risk" based on the registered information.

[0110] If the state of the object is highly dangerous (Yes in S32), it is determined that changing the execution of the process for the host vehicle is dangerous, and the reception unit 14 stops accepting the request (S33). If the state of the object is not highly dangerous (No in S32), the change in the execution of the process for the host vehicle is permitted, and the reception unit 14 continues accepting the request (S34). As a result, accidents caused by driver error can be reduced in situations where it is necessary to stop or slow down reliably. Furthermore, it is possible to reduce the processing load within the device in situations where it is not necessary to stop or slow down reliably.

[0111] Furthermore, an external server may receive information such as the type of object, the state of the object, and the vehicle position when the "cancellation of execution of the predetermined process" is cancelled from multiple vehicles. In this case, the external server estimates the conditions for canceling the "cancellation of execution of the predetermined process" through statistical processing. The external server updates the map data based on the estimated conditions and distributes the estimated conditions to the vehicles. When the on-board device of the vehicle to which the conditions have been distributed detects an object that matches the conditions, the reception unit 14 stops accepting the input of "cancellation of execution of the predetermined process (first process)."

[0112] Furthermore, the display of an area where an input to cancel the execution of a predetermined process due to the state of the object can be made may be controlled (highlighted, superimposed with a marker, changed in color, etc.).

[0113] In addition, in the case of a system consisting of an unmanned vehicle such as an unmanned taxi or bus and a monitoring center that monitors the running of the unmanned vehicle, the output unit and input unit are located in the monitoring center, and the rest are located in the unmanned vehicle. In this case, an input for changing the processing due to the cause is received from the input unit of the monitoring center.

[0114] Although the embodiments and examples have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0115] This application claims priority based on Japanese Patent Application No. 2016-193932, filed September 30, 2016, the disclosure of which is incorporated herein by reference in its entirety.

Claims

1. a monitoring unit that recognizes objects around the vehicle and the states of the objects based on outputs of external sensors mounted on the vehicle; a vehicle control unit that controls the vehicle in accordance with the object and the state; an output control unit that causes an output device to output the content of the control and cause information indicating at least one of the object and the state that caused the control while the control is being executed; a receiving unit that receives an input corresponding to the cause information, the input being for changing the control that is being executed; An in-vehicle device having the above.

2. The in-vehicle device according to claim 1 , wherein the receiving unit receives an input to cancel the control as the input for changing the control.

3. the receiving unit receives an input for correcting a result of the recognition as an input for changing the control; The in-vehicle device according to claim 1 , wherein the vehicle control unit controls the vehicle based on the corrected result of the recognition.

4. the vehicle control unit decelerates or stops the vehicle in accordance with the object and the state recognized by the monitoring unit, the output control unit causes the output device to output the cause information indicating at least one of the object and the state that caused the deceleration or stop; The in-vehicle device according to claim 3 , wherein the vehicle control unit cancels the control of decelerating or stopping the vehicle in response to an input for correcting the recognition result.

5. the output control unit causes the output device to output the plurality of pieces of cause information that caused the control; The in-vehicle device according to claim 1 , wherein the accepting unit accepts an input for changing the control caused by each of the plurality of pieces of cause information, in response to each of the plurality of pieces of cause information.

6. the control is a stop of movement of the vehicle; the output control unit causes the output device to output the plurality of pieces of cause information that caused the vehicle to stop; The in-vehicle device according to claim 5 , wherein the accepting unit accepts an input for canceling the stop of the vehicle caused by each of the plurality of pieces of cause information, in response to each of the plurality of pieces of cause information.

7. The in-vehicle device according to claim 6 , wherein the vehicle control unit, upon receiving an input for releasing the vehicle from a stopped state in response to all of the plurality of pieces of cause information that caused the vehicle to stop, releases the vehicle from a stopped state.

8. 8. The in-vehicle device according to claim 6, wherein the vehicle control unit does not release the vehicle from the stopped state if it receives an input to release the vehicle from the stopped state corresponding to some of the plurality of pieces of cause information that caused the vehicle to stop, but does not receive an input to release the vehicle from the stopped state corresponding to other pieces of cause information.

9. The vehicle control unit 9. The in-vehicle device according to claim 1, wherein, after the receiving unit receives an input to cancel a first control caused by a first state of a first object, when the monitoring unit detects a predetermined movement of the first object, the in-vehicle device executes the first control again.

10. The vehicle control unit 10. The in-vehicle device according to claim 1, wherein, after the receiving unit receives an input to cancel a second control caused by a second state of a second object, when the monitoring unit detects a predetermined movement of the second object, the second control is executed again, with the second state of the second object being the cause of the control.

11. a determination unit that determines the monitoring accuracy of the monitoring unit, The in-vehicle device according to claim 1 , wherein the reception unit stops receiving an input for changing the control when the monitoring accuracy is high.

12. The computer a monitoring step of recognizing objects and states of the objects around the vehicle based on outputs of external sensors mounted on the vehicle; a vehicle control step of controlling the vehicle in accordance with the object and the state; an output control step of causing an output device to output, during execution of the control, the content of the control and cause information indicating at least one of the object and the state that caused the control; a receiving step of receiving an input corresponding to the cause information for changing the control being executed; A control method for performing the above.

13. Computer, a monitoring means for recognizing objects around the vehicle and the state of the objects based on the output of an external sensor mounted on the vehicle; a vehicle control means for controlling the vehicle in accordance with the object and the state; an output control means for causing an output device to output the content of the control and cause information indicating at least one of the object and the state that caused the control while the control is being executed; a receiving means for receiving an input corresponding to the cause information, the input being for changing the control being executed; A program that functions as a

Images