DRIVING ASSISTANCE DEVICE, DRIVING ASSISTANCE SYSTEM, DRIVING ASSISTANCE METHOD, AND AUTOMATIC DRIVING VEHICLE

The driving assistance system addresses the complexity of existing automatic driving interfaces by enabling intuitive gesture-based commands, improving safety and accessibility through a human-machine interface that simplifies vehicle operation.

DE112016003498B4Active Publication Date: 2024-12-05PANASONIC AUTOMOTIVE SYST CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
DE112016003498
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2015-07-31
Filing Date
2016-07-19
Publication Date
2024-12-05
Estimated Expiration
2036-07-19

AI Technical Summary

Technical Problem

Existing automatic driving systems require complex and non-intuitive user interfaces for issuing driving commands, leading to potential misoperations and limiting accessibility for drivers with reduced abilities or those unfamiliar with advanced driving features.

Method used

A driving assistance system that utilizes a human-machine interface (HMI) for intuitive gesture-based commands, allowing drivers to control vehicle operations through touch and gesture inputs on a display, enabling direct interaction with vehicle icons to execute commands such as lane changes, overtaking, and route adjustments.

Benefits of technology

Facilitates safer and more accessible automatic driving by reducing the likelihood of misoperations and allowing drivers to conveniently issue commands without the need for manual controls, thereby enhancing user experience and safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Driving assistance device, including: an image output unit that outputs an image containing a vehicle object representing a vehicle and the vehicle's surroundings to a display unit; an operating signal input unit which, when a receive mode of the driving assistance device is set as a receive release mode in which operation by a user can be received, receives a gesture control that moves the vehicle object in the image displayed on the display unit to a position of an object that represents a predetermined target other than the vehicle; and a command output unit which, when the user's gesture control is received, issues a first command ordering a change in a relative positional relationship between the vehicle and the predetermined destination, or a second command ordering the output of a message from the vehicle to the predetermined destination, to an automatic driving control unit that controls automatic driving, where the receiving mode is changed from the receiving release mode to a receiving lock mode, in which no operation by the user can be received if the first or second command is present and a first period of time has passed after the determination of the first or second command based on gesture operation.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present invention relates to a technique that supports a driver's driving control command to a vehicle during automated driving. GENERAL STATE OF THE ART

[0002] DE 10 2010 030 463 A1 relates to a method for assisting a driver of a motor vehicle during a driving maneuver, in which, in a first step, the environment of the motor vehicle is detected and a two-dimensional representation of the environment of the motor vehicle is displayed, in a second step a desired end position of the vehicle is entered by the driver, whereby the end position is marked in the two-dimensional representation, and in a final step a trajectory to reach the end position is determined and instructions are given to the driver to follow the trajectory or an automatic driving maneuver is carried out in which the vehicle is maneuvered along the trajectory to the end position.

[0003] DE 10 2011 087 459 A1 relates to a driver assistance system for a motor vehicle, comprising a sensing device for detecting the vehicle's surroundings and a user interface operable by the driver. The driver assistance system is designed such that the driver can specify one or more objects in the environment detected by the sensing device via the user interface, whereupon one or more functions of the driver assistance system are executed for the specified object(s).

[0004] A method according to DE 10 2011 121 746 A1 for navigating a vehicle using a graphic projection display comprises monitoring a navigation state graph representing a navigation intention displayed on the graphic projection display, monitoring a user input specified on a section of the graphic projection display, initiating a user-defined navigation command based on the monitored navigation state graph and the monitored user input, and operating the vehicle according to the user-defined navigation command.

[0005] In recent years, there has been significant progress in the development of automated driving. Regarding automated driving, the levels of autonomy defined by the National Highway Traffic Safety Administration (NHTSA) in 2013 are divided into "no autonomy" (Level 0), "autonomy for certain functions" (Level 1), "autonomy for complex functions" (Level 2), "semi-autonomous driving" (Level 3), and "fully autonomous driving" (Level 4). Level 1 is a driver assistance system that automatically performs one of the following functions: acceleration, deceleration, and steering. Level 2 is a driver assistance system that automatically performs two or more of these functions in coordination. In all cases, the driver must remain involved in operating the vehicle.Level 4 autonomy is a fully automated driving system that handles all aspects of acceleration, deceleration, and steering automatically, meaning the driver is not involved in driving. Level 3 autonomy is a semi-automated driving system that also handles all aspects of acceleration, deceleration, and steering automatically, but the driver may still intervene if necessary.

[0006] One form of automated driving is one in which the driver does not operate an existing driving control unit such as a steering wheel, accelerator pedal, or the like, but instead commands the vehicle to perform a specific driving action, such as changing lanes, overtaking, following another vehicle, or similar maneuvers. This form requires a user interface that minimizes the risk of user error. List of literature on patent literature

[0007] PTL 1: Japanese Patent Publication No. JP H10 - 105 885 A BRIEF SUMMARY OF THE INVENTION

[0008] One object of the present invention is to provide a technique that allows a driver to intuitively and conveniently command a vehicle to perform a specific driving operation.

[0009] To solve the problems described above, a driving assistance device according to one aspect of the present invention has the features according to claim 1.

[0010] Another aspect of the present invention is also one in which the present invention is effectively transformed into a device, a system, a method, a program, a recording medium on which a program is recorded, and a vehicle equipped therewith by appropriately combining the above structural elements.

[0011] According to the present invention, the driver can intuitively and conveniently command a vehicle to perform a specific driving operation. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram illustrating the construction of a vehicle according to an embodiment of the present invention. Fig. Figure 2 is a diagram showing an example of a basic sequence of a detection unit, an automatic driving control device, an HMI control device, a display unit, and an input unit in Fig. 1 illustrates. Fig. Figure 3 is a diagram that shows an example of a basic flowchart to explain the processing in the HMI control device in Fig. 1 illustrates. Fig. Figure 4 is a flowchart explaining the processing for updating the receiving mode. Fig. 5 is a flowchart that shows an example of the processing for command determination in the case of inputting a gesture control in step S9 of Fig. 3 illustrated. Fig. Figure 6 is a diagram illustrating an example of an automatic transmission control command table. Fig. Figure 7 is a flowchart illustrating an example of a gesture control that drags a vehicle icon to another lane. Fig. Figure 8 is a diagram that shows an example of gesture control according to the flowchart of Fig. 7 illustrates. Fig. Figure 9 is a flowchart illustrating an example of a gesture control that flicks a personal vehicle icon. Fig. 10A is a diagram that shows an example of gesture control according to the flowchart of Fig. 9 illustrates. Fig. 10B is a diagram that shows an example of gesture control according to the flowchart of Fig. 9 illustrates. Fig. Figure 11 is a flowchart illustrating an example of a gesture control that touches two points on a self-propelled vehicle icon and a third-party vehicle icon and rotates the two points. Fig. 12 is a diagram that shows an example of gesture control according to the flowchart of Fig. 11 illustrated. Fig. Figure 13 is a flowchart illustrating an example of a gesture control that drags a self-contained vehicle icon onto a third-party vehicle icon. Fig. 14 is a diagram that shows an example of gesture control according to the flowchart of Fig. 13 illustrated. Fig. Figure 15 is a diagram illustrating an example of a gesture control which, while following another vehicle, touches two points on a self-operated vehicle icon and a foreign vehicle icon and increases / decreases the distance between the self-operated vehicle icon and the foreign vehicle icon. Fig. 16 is a diagram that shows an example of gesture control according to the flowchart of Fig. 15 illustrated. Fig. Figure 17 is a flowchart illustrating an example of gesture control, which, while following another vehicle, touches two points on a self-contained vehicle icon and a foreign vehicle icon and rotates the two points. Fig. 18 is a diagram that shows an example of gesture control according to the flowchart of Fig. 17 illustrates. Fig. Figure 19 is a flowchart illustrating an example of a gesture control that swipes a vehicle icon. Fig. 20 is a diagram that shows an example of gesture control according to the flowchart of Fig. 19 illustrated. Fig. Figure 21 is a flowchart illustrating an example of dragging a personal vehicle icon onto a specific area (a road signpost). Fig. 22 is a diagram that shows an example of gesture control according to the flowchart of Fig. 21 illustrated. Fig. 23 is a flowchart that provides an example of the processing of drawing a predictive trajectory / route candidate in step S9e of Fig. 5 illustrates. Fig. 24 is a flowchart that shows another example of the processing of drawing a predictive trajectory / route candidate in step S9e of Fig. 5 illustrates. Fig. 25A is a diagram that shows an example of gesture control according to the flowchart of Fig. 24 illustrated. Fig. 25B is a diagram that shows an example of gesture control according to the flowchart of Fig. 24 illustrated. Fig. 25C is a diagram that shows an example of gesture control according to the flowchart of Fig. 24 illustrated. Fig. Figure 26 is a diagram illustrating an example of an error screen display. Fig. Figure 27 is a diagram illustrating a display example during a period in which a self-contained vehicle icon is dragged and dropped onto a third-party vehicle icon. Fig. Figure 28 is a diagram illustrating another example of how a vehicle icon is dragged and dropped onto a vehicle icon belonging to another vehicle during a period of time. Fig. 29 is a diagram illustrating an example of how to display a drawn vehicle icon when there is an area where it is not possible to place it, or the like. Fig. 30A is a diagram illustrating another example of how to display a drawn own vehicle icon when there is an area where no drop-off is possible, or the like. Fig. 30B is a diagram illustrating another example of how to display a drawn own vehicle icon when there is an area where it is not possible to drop it, or the like. Fig. 30C is a diagram illustrating another example of how to display a drawn own vehicle icon when there is an area where dropping is not possible, or the like. Fig. 31 is a diagram illustrating a display example when no drag operation is possible. Fig. 32A is a diagram illustrating a different display example when pull operation is not possible. Fig. Figure 32B is a diagram illustrating a different display example when drag operation is not possible. Fig. 33 is a diagram illustrating a first example of a control operation that is carried out while a self-contained vehicle icon is being placed and the own vehicle is completing an overtaking maneuver. Fig. Figure 34 is a diagram illustrating a second example of a control operation performed while a self-contained vehicle icon is being placed and the own vehicle is completing an overtaking maneuver. Fig. Figure 35 is a diagram illustrating a third example of a control operation performed while a self-contained vehicle icon is being placed and the own vehicle is completing an overtaking maneuver. Fig. Figure 36 is a diagram illustrating a fourth example of a control being performed while a self-carriage icon is being placed and the self-carriage is completing an overtaking maneuver. DESCRIPTION OF EXECUTION FORMS

[0012] Before describing an exemplary embodiment of the present invention, problems with a prior art device will be briefly described. If only road conditions are displayed, the driver must, when commanding a specific driving operation to the vehicle, individually operate each switch, turn a steering wheel, or press an accelerator pedal while checking the road conditions. Furthermore, the vehicle's line of sight moves. Therefore, there is a possibility of incorrect operation.Furthermore, in a system where each switch is assigned to a specific driving operation, such as changing lanes, overtaking, following, or the like, the correlation between a switch operation and the control of the automatic driving is not intuitive. Selecting a command from many switches, or choosing a function that changes depending on the situation with a reduced number of switches, becomes a complex task. Therefore, the problem is that a broad user group, including those who have never driven before, those who wish to continue driving despite a decline in their driving skills, and others, cannot use an autonomously driving vehicle without practice.

[0013] With reference to the drawings, embodiments of the present invention will be described below. Fig. Figure 1 is a block diagram illustrating the structure of a vehicle 1 according to an embodiment of the present invention and illustrating a structure related to automatic driving. The vehicle 1 (a separate vehicle) with an automatic driving mode comprises a driving assistance device (a human-machine interface (HMI) control device) 10, an automatic driving control device (automatic driving controller) 20, a display device 30, a detection unit 40, and a driving operating unit 50.

[0014] The display device 30 comprises a display unit 31 and an input unit 32. The display device 30 can be a head unit such as a vehicle navigation system, a display audio system or the like, a portable device such as a smartphone, a tablet or the like, or a dedicated console device.

[0015] The display unit 31 is a liquid crystal display, an organic electroluminescent (EL) display, or a head-up display (HUD). The input device 32 is a user interface that receives input from a user. The display unit 31 and the input unit 32 can be an integrated touch panel display. A device that receives gesture input at a position at a predetermined distance from the touch panel can be used, such as a proximity touch panel capable of detecting the approach of a hand over a touch panel or touchpad, or the position of a finger as a result of hovering manipulation. The input unit 32 can include an input device such as a mouse, a stylus, a control ball, or the like to support gesture input. A pen that emits visible or infrared light can also be used.

[0016] The display unit 31 and the input unit 32 can be physically separate from each other instead of forming an integrated touch panel display. For example, the input unit 32 may have a sensor such as a camera or the like, and it may be a contactless input device that allows gesture control in the air. For example, an operating procedure is envisioned in which a drag is initiated by pointing at a target with one finger and bringing the thumb and index finger together and closing, and the drag is ended by a gesture in which the thumb and index finger separate.

[0017] The driving assistance device 10 and the display device 30 can be connected by wired communication such as a dedicated line, a control unit network (CAN) or the like, or can be connected by wired or wireless communication such as USB, Ethernet (registered trademark), WiFi (registered trademark), Bluetooth (registered trademark) or the like.

[0018] The detection unit 40 comprises a position information acquisition unit 41, a sensor 42, a speed information acquisition unit 43, and a plan information acquisition unit 44. The position information acquisition unit 41 obtains the current position of the vehicle 1 from a Global Positioning System (GPS) receiver. The sensor 42 is a general term for various sensors used to detect conditions outside the vehicle and the state of the vehicle 1. For example, a camera, millimeter-wave radar, LIDAR (laser image detection and ranging), a temperature sensor, a pressure sensor, a humidity sensor, a light sensor, or the like may be mounted as a sensor for detecting conditions outside the vehicle.Conditions outside the vehicle could include a traffic situation in which the vehicle is moving, environmental conditions including weather, and conditions around the vehicle, such as the position or state of another vehicle moving in the vicinity. Any information external to the vehicle that can be detected by the sensor is conceivable. For example, an accelerometer, gyroscope, geomagnetic sensor, tilt sensor, or similar sensor could be mounted to detect the state of vehicle 1. The speed information acquisition unit 43 obtains the current speed of vehicle 1 from a vehicle speed sensor. The map information acquisition unit 44 obtains map information around the current position of vehicle 1 from a map database.The plan database can be recorded on a recording medium in vehicle 1 or, when used over a network, downloaded from a plan server.

[0019] The detection unit 40 and the automatic driving control device 20 are connected to each other via wired communication, such as a dedicated line, USB, Ethernet (registered trademark), a control unit network (CAN), or the like. A configuration can be used in which data acquired and detected by the detection unit 40 is directly output by the detection unit 40 to the driving assistance device 10.

[0020] The driving unit 50 comprises a steering system 51, a brake pedal 52, an accelerator pedal 53, and a turn signal switch 54. In an automatic driving mode according to the present embodiment, acceleration, deceleration, steering, and the flashing of the turn signal are objectives for automatic control by the automatic driving control device 20. Fig. Figure 1 shows a control unit for manually performing these controls. Information can be output to the driving assistance device 10 indicating that the driving unit 50 has been moved slightly by hand by the driver.

[0021] The steering unit 51 is a control unit for steering the vehicle. When the steering unit 51 is turned by the driver, the vehicle's direction of travel is controlled via a steering actuator. The steering actuator can be electronically controlled by an electronic steering control unit (steering ECU).

[0022] The brake pedal 52 is a control unit for decelerating the vehicle 1. When the brake pedal 52 is pressed by the driver, the vehicle is decelerated via a brake actuator. The brake actuator can be electronically controlled by a brake ECU.

[0023] The accelerator pedal 53 is a control unit for accelerating the vehicle 1. When the accelerator pedal 53 is pressed by the driver, an engine speed and / or an electric motor speed are controlled via an accelerator pedal actuator. In a purely gasoline-powered vehicle, the engine speed is controlled. In a purely electric vehicle, the electric motor speed is controlled. In a hybrid vehicle, both the engine speed and the electric motor speed are controlled. The accelerator pedal actuator can be electronically controlled by at least one engine ECU and one electric motor ECU.

[0024] The turn signal switch 54 is a control unit used to flash a turn signal, thus indicating the vehicle's direction. When the turn signal switch 54 is turned on or off by the driver, the turn signal is activated or deactivated via a turn signal control device. The turn signal control device includes a drive circuit, such as a relay, that controls the power supply to a turn signal bulb, or the like.

[0025] All components from the steering ECU, brake ECU, engine ECU, electric motor ECU, turn signal control unit, and automatic transmission control unit 20 are interconnected via wired communication such as CAN, a dedicated line, or the like. Each component from the steering ECU, brake ECU, engine ECU, electric motor ECU, and turn signal control unit sends a status signal to the automatic transmission control unit 20, indicating the status of each component from the steering, brake, engine, electric motor, and turn signal.

[0026] In automatic driving mode, each of the steering ECU, brake ECU, engine ECU, and electric motor ECU drives its corresponding actuator according to a control signal supplied by the automatic driving control device 20. In manual driving mode, a configuration can be used in which a command is transmitted mechanically directly from each of the steering 51, brake pedal 52, and accelerator pedal 53 to the corresponding actuator, or a configuration in which an electronic control device intervenes via the corresponding ECU. The turn signal control device switches the turn signal on / off according to the control signal supplied by the automatic driving control device 50 or a command signal from the turn signal switch 54.

[0027] The automatic driving control device 20 is an automatic driving control device that has an automatic driving control function and comprises a control unit 21, a memory unit 22, and an input / output unit 23. The structure of the control unit 21 can be implemented through interaction between hardware and software resources or solely through hardware resources. Hardware resources can include a processor, read-only memory (ROM), random-access memory (RAM), and other low-level interface (LSI) circuits, while software resources can include an operating system, an application, and a program such as firmware or the like. The memory unit 22 comprises a non-volatile recording medium such as flash memory or the like. The input / output unit 23 performs various communication controls according to different communication formats.

[0028] The control unit 21 calculates a control value for controlling an automatic control objective, such as the direction of travel of the vehicle 1, by applying various parameter values ​​collected by the detection unit 40 and various ECUs to an automatic driving algorithm. The control unit 21 transmits the calculated control value to the ECU or control device of each control objective. In the present embodiment, the control value is transmitted to the steering ECU, the brake ECU, the engine ECU, and the turn signal control device. In the case of an electric vehicle or a hybrid vehicle, the control value is transmitted to the electric motor ECU instead of, or in addition to, the engine ECU.

[0029] The driving assistance device 10 is a human-machine interface (HMI) control device for performing an interface function between the vehicle 1 and the driver and comprises a destination unit 11, an image generation unit 12, a command unit 13, and an input / output unit 14. The destination unit 11, the image generation unit 12, and the command unit 13 can be implemented through interaction between hardware and software resources or solely by hardware resources. Hardware resources can include a processor, ROM, RAM, and other LSI circuits, while software resources can include an operating system, an application, and a program such as firmware or the like. The input / output unit 14 performs various communication controls according to different communication methods.The input / output unit 14 comprises an image output unit 14a, an operating signal input unit 14b, a command output unit 14c, and a vehicle information input unit 14d. The image output unit 14a outputs an image generated by the image generation unit 12 to the display unit 31. The operating signal input unit 14b receives an operating signal, which is entered by the driver, a passenger, or a user outside the vehicle via the input unit 32, and outputs the operating signal to the destination unit 11. The command output unit 14c outputs the command issued by the command unit 13 to the automatic driving control device 20.The vehicle information input unit 14d receives detection data obtained by the detection unit or vehicle information generated by the automatic driving control device and outputs the detection data or the vehicle information to the image generation unit 12.

[0030] The automatic driving control device 20 and the HMI control device 10 are directly connected to each other via a signal line. Alternatively, a configuration can be used in which the automatic driving control device 20 and the HMI control device 10 are connected to each other via a CAN bus. Another configuration can be used in which the automatic driving control device 20 and the HMI control device 10 are combined into a single control device.

[0031] Fig. Figure 2 is a diagram showing an example of a basic sequence of the detection unit 40, the automatic driving control device 20, the HMI control device 10, the display unit 31, and the input unit 32. Fig. Figure 1 illustrates this. The detection unit 40 detects vehicle position information, vehicle path information, and vehicle environment status and outputs the detected information to the automatic driving control unit 20 (P1). The automatic driving control unit 20 outputs the vehicle position information, vehicle path information, and vehicle environment information obtained from the detection unit 40 to the HMI control unit 10 (P2). Based on the information obtained from the automatic driving control unit 20, the HMI control unit 10 generates a schematic diagram containing the vehicle and its environment status (P3). The HMI control unit 10 outputs the generated schematic diagram to the display unit 30, and the schematic diagram is displayed on the display unit 31 (P4).

[0032] A user viewing the schematic diagram displayed on the display unit 31 touches the input unit 32 (P5). The input unit 31 outputs coordinate data of the position where the touch is detected to the HMI control device 10 (P6). Based on the coordinate data obtained from the display unit 30, the HMI control device 10 determines the type of command (P7). The HMI control device 10 accepts additional inputs until a certain period of time has elapsed (P8 to P12). After determining the command, the HMI control device 10 generates another schematic diagram indicating that the command is being issued (P8). The HMI control device 10 outputs the regenerated schematic diagram to the display unit 30, and the regenerated schematic diagram is displayed on the display unit 31 (P9).If the command corresponding to gesture control by touch by the user does not exist, the HMI control device 10 generates a schematic diagram containing an error message and outputs the schematic diagram to the display device 30, and the schematic diagram is displayed on the display unit 31.

[0033] When a user, viewing the schematic diagram indicating that the command is being issued, touches the input unit 32 (P10), the display unit 31 outputs the coordinate data of the position where the touch is detected to the HMI control unit 10 (P11). Based on the coordinate data obtained from the display unit 30, the HMI control unit 10 performs additional command processing (P12). If no new command is entered during this additional command processing (P12), the HMI control unit 10 outputs the command determined at P7 to the automatic drive control unit 20 (P13 and P14). If a new command is entered during this additional command processing (P12), the HMI control unit 10 outputs a new command to the automatic drive control unit 20. If the entered new command is a cancel command, the HMI control unit 10 cancels the command output.The automatic driving control device 20 can perform processing to overwrite and cancel the original command with a new command. In this case, the HMI control device 10 sends the command to the automatic driving control device 20 after the command determination processing at P7 and P12, and the automatic driving control device 20 performs the overwriting and canceling processing according to an internal state of the automatic driving control device 20.

[0034] The detection unit 40 periodically detects the vehicle's position information, its route information, and its surroundings and outputs this information to the automatic driving control unit 20 (P15). Based on this information, the automatic driving control unit 20 determines whether the control command issued by the HMI control unit 10 is executable or not (P16). If it is determined that the control is executable, the automatic driving control unit 20 outputs a control start message to the HMI control unit 10 (P17). Upon receiving the control start message, the HMI control unit 10 again generates a schematic diagram containing a message indicating that the control is being carried out (P18).The HMI control device 10 outputs the regenerated schematic diagram to the display device 10, and the regenerated schematic diagram is displayed on the display unit 31 (P19). Although no corresponding display has been made, the automatic driving control device 20 calculates a control value for controlling the driving unit 50, which executes the issued command, by applying various parameter values ​​collected by the detection unit 40 or various ECUs to an automatic driving algorithm, and sends the control value to the ECU or control device of each control target. The control value is a specific value used to automatically control the control target, such as the direction of travel of the vehicle 1 or the like. The driving unit 50 operates based on the specified control value.If a predetermined control value or detection data obtained by the detection unit 40 is a specific value (within a predetermined range), and if the automatic driving control device 20 determines that a condition of the issued command is met, the driving operating unit 50 determines that the execution of the command is complete.

[0035] Upon receiving a control completion message from the automatic drive control device 20, the HMI control device 10 generates a schematic diagram containing a message indicating that the control operation is complete and outputs the generated schematic diagram to the display device 30. During a period in which no user input is received, the HMI control device 10 generates a schematic diagram containing a message indicating that no input is being received and outputs the generated schematic diagram to the display device 30.

[0036] Fig. Figure 3 is a diagram that shows an example of a basic flowchart to explain the processing of the HMI control device 10 in Fig. Figure 1 illustrates this. The determination unit 11 of the HMI control device 10 checks whether a driving mode is an automatic driving mode or a manual driving mode (S1). In the case of manual driving mode (N in S2), the process is terminated. In the case of automatic driving mode (Y in S2), processing is carried out as follows.

[0037] The sensor information input by the detection unit 40 to the automatic driving control device 20 is updated as necessary (S3). Based on the vehicle's own position information, its own route information, and its own environment information input by the automatic driving control device 20, the image generation unit 12 of the HMI control device 10 generates a schematic diagram containing the vehicle and its surroundings, and displays the generated schematic diagram on the display unit 31 (S4). The determination unit 11 checks whether a receive mode is a receive enable mode, in which operation by a user can be accepted, or a receive block mode, in which no operation by the user can be accepted (S5).If the receive mode is receive lock mode (N in S6), the process terminates. If the receive mode is receive release mode (Y in S6), the destination unit 11 determines whether or not the user has touched input unit 32 (S7). If the user has not touched input unit 32 (N in S8), processing is performed to determine the elapsed time, as described later. If the user has touched input unit 32 (Y in S8), the destination unit 11 determines a control command corresponding to the user's gesture input (S9). The details of the processing to determine the control command will be discussed later.

[0038] If the control command determined in step S9 is not a cancellation command (N in S10), the image generation unit 12 indicates to the display unit 31 that the command has been issued (S11). If a predetermined time has elapsed after the control command has been determined (Y in S12), and a control command determined in step S9 is present (Y in S13), the display unit 31 indicates that the acceptance of operations is blocked (S14), the acceptance mode is updated by the determination unit 11 from acceptance release mode to acceptance block mode (S15), and the determined control command is issued by the command unit 13 to the automatic driving control device 20 (S16). Until the predetermined time has elapsed (N in S12), the process proceeds to step S3.

[0039] If the specified control command is a cancellation command (Y in S10), the cancellation is displayed in step S10 (S110) and the process terminates. If no control command specified in step S9 is present in step S13 (N in S13), an input error is displayed (S111) and the process terminates. The automatic driving control unit 10 periodically detects the vehicle's position information, its route information, and its surroundings from the detection unit 40. Since the surroundings of the vehicle are constantly changing, it may happen that after the control command is issued to the automatic driving control unit 20, it is determined that the control command cannot be executed. For example, after a command to follow another vehicle, it is possible that other vehicles have positioned themselves between the vehicle and the other vehicle.If it is determined that the control command can be executed by the automatic driving control device 20 (Y in S17), the image generation unit 12 indicates to the display unit 31 that the control is being carried out (S18) and begins a timer by activating a timer (S19). If it is determined that the execution of the control command by the automatic driving control device 20 is not possible (N in S17), the image generation unit 12 indicates the error that control is not possible to the display unit 31 (S112).

[0040] Fig. Figure 4 is a flowchart explaining the processing for updating the receive mode. When the timer count reaches a set value (for example, 10 seconds) (Y in S113), the destination unit 11 of the HMI control device 10 updates the receive mode from the receive lock mode to the receive enable mode (S114). The timer count can be changed depending on the ambient state of the vehicle. The destination unit 11 can also update the receive mode from the receive lock mode to the receive enable mode if it receives a message from the automatic driving control device 20 indicating the completion of the control operation, or if, based on the behavior of the vehicle 1, it is determined that the control operation corresponding to the control command is complete.

[0041] Fig. 5 is a flowchart that shows an example of the processing for command determination in step S9 in Fig. Figure 3 illustrates this in the case of gesture input. When a user begins touching the input unit 32 (Y in S9a), the image generation unit 13 draws an area in which placement is possible in the schematic diagram and displays the area on the display unit 31 (S9b).

[0042] The determination unit 11 receives a contact event (S9c) generated in the input unit 32 and determines the type of contact event (S9d). If the type of contact event is the start of contact / movement (a start of contact / movement in S9d), the image generation unit 12 draws a predicted path / route candidate of vehicle 1 in the schematic diagram, and the predicted path / route candidate of vehicle 1 is displayed on the display unit 31 (S9e). The process then returns to step S9c.

[0043] If the type of touch event is an end of touch (an end of touch in S9d), the determination unit 11 determines the type of gesture control. Typical gesture controls include pull-down, flick, swipe, pull-up / pull-down, and rotary. The determination unit 11 determines the type of gesture control based on a time series transition in the touch event (S9f). Specifically, pull-down, flick, and swipe controls are distinguished based on the dwell time at the starting point of the touch, the movement speed after the start of the movement, and the dwell time at the end point of the touch. If the touch points are two points and the distance between the two points changes by a predetermined distance or more, a pull-up / pull-down control is determined.If the distance between the two points does not change by a predetermined distance or more, and the positions of the two points are changed, a rotary control is determined.

[0044] The determination unit 11 uses the number of touch points, the starting position of the touch, the ending position of the operation, and the type of gesture operation as a key to refer to an automatic driving control command table (S9g). If a control command corresponding to four keys is present in the automatic driving control command table (Y in S9h), the determination unit 11 determines that the control command is a control command for output to the automatic driving control device 20 (S9i). If a control command corresponding to four keys is missing in the automatic driving control command table (N in S9h), the image generation unit displays an error message on the display unit 31 (S9j).

[0045] Fig. Figure 6 is a diagram illustrating an example of an automatic vehicle control command table 11a. The automatic vehicle control command table 11a is incorporated into the determination unit 11. An example of the present embodiment is described below, in which the position of the start of contact is a vehicle marking in the schematic diagram.

[0046] The in Fig. Illustrated automatic driving control command table 11a defines a foreign vehicle marking, an emergency vehicle marking, a pedestrian marking, a lane, a shoulder, an intersection, and a specific area as the positions of the end of the touch. Gesture controls are defined as a pull-down operation, a flick / swipe operation, a pull / pull-up operation, and a turn operation.

[0047] When a user drags and drops their own vehicle marker onto the other vehicle marker, a command is issued to follow / drive in the lane. When a user drags and drops their own vehicle marker onto the emergency vehicle marker (an ambulance or similar) and the emergency vehicle marker is positioned to the right of the own vehicle marker, a command is issued to temporarily stop to the left. If the emergency vehicle marker is positioned to the left of the own vehicle marker, a command is issued to temporarily stop to the left. When the user drags and drops their own vehicle marker onto the pedestrian marker, a command is issued to stop next to the pedestrian.

[0048] If the user drags the vehicle marker to a different lane and the marker is not positioned near an intersection, a command is issued ordering a lane change to that lane. If the vehicle marker is positioned near an intersection, and the lane where it is placed is a right-turn lane, a command is issued ordering a lane change to the right-turn lane and a right turn. If the vehicle marker is positioned near an intersection, and the lane where it is placed is a left-turn lane, a command is issued ordering a lane change to the left-turn lane and a left turn.

[0049] If the user drags and drops the vehicle marker onto a road shoulder, a command is issued ordering the vehicle to park on the shoulder. If the user drags and drops the vehicle marker onto the entrance of a lane after a right turn at an intersection, a command is issued ordering the vehicle to turn right. If a user drags and drops the vehicle marker onto the entrance of a lane after a left turn at an intersection, a command is issued ordering the vehicle to turn left.

[0050] If the user drags and drops the vehicle marker onto a specified area other than a lane, intersection, or shoulder, a command corresponding to that specified area is issued. For example, a command ordering temporary swerving onto the specified area is issued. If the specified area is an area on a road signpost displaying a place name or area name, a command ordering a route change to a destination corresponding to that place name or area name is issued. A command ordering the setting of a transit point corresponding to that place name or area name may also be issued. If the specified area is an area on a road signpost displaying a street or road designation, a command ordering a route change to a route on that street is issued.If the designated area is one where an exit sign for a toll road is displayed, a command will be issued instructing the vehicle to leave the toll road via the exit.

[0051] When the user flicks or swipes the vehicle marker, a command is issued that changes the vehicle speed according to the direction and distance traveled by the marker. For example, the vehicle speed can be set to increase with an upward swipe, decrease with a downward swipe, and change more significantly with a greater distance traveled. Flicking or swiping the vehicle marker can also issue a command that changes the distance between the user's vehicle and a vehicle ahead, according to the direction and distance traveled by the marker.For example, the distance between the vehicle and the vehicle in front can be set so that it is reduced by an upward swipe, increased by a downward swipe, and changed more drastically the greater the distance traveled.

[0052] If the user flicks or swipes the vehicle marker when another lane is present, a command to change lanes to that lane can be issued. If the user flicks or swipes the vehicle marker toward a right-turn lane near an intersection, a command to change lanes to the right-turn lane and turn right will be issued. If the vehicle marker is flicked or swiped toward a left-turn lane near an intersection, a command to change lanes to the left-turn lane and turn left will be issued.

[0053] When the user touches two points on the vehicle marker and pulls the marker in or out, a command is issued that orders a change in speed corresponding to the change in the distance between the two points. For example, the vehicle speed can be set so that it decreases when pulling the marker in, increases when pulling it out, and changes more significantly as the distance between the two points increases.

[0054] When the user touches two points on their own vehicle's marker and another vehicle's marker, and then adjusts the distance between the markers while their vehicle and the other vehicle are driving one behind the other, a command is issued that orders a change in the distance between the two vehicles. For example, the distance between the user's vehicle and the other vehicle can be set so that it decreases when the marker is pulled down, increases when it is pulled up, and the change is more pronounced as the distance between the two points increases.If the user touches two points on their own vehicle's marker and another vehicle's marker, and then adjusts the distance between the markers while their vehicle and the other vehicle are not driving directly behind each other, a command is issued that orders a normal change in the distance between the vehicle and the other vehicle. In this case, the control system for maintaining a constant distance will not operate even if the distance between the vehicle and the other vehicle changes over time.

[0055] If the user touches two points on the vehicle's marker and rotates them while the vehicle is near an intersection, a command is issued that orders a right turn or a left turn, depending on the direction the two points are rotated. If a user touches two points on the vehicle's marker and rotates them while the vehicle is near a parking space, a command is issued that orders the vehicle to change direction according to the direction the two points are rotated.

[0056] If the user touches two points on their own vehicle's marker and another vehicle's marker and rotates these two points while their own vehicle and the other vehicle are following each other, a command is issued ordering the user to swap the positions of their own vehicle and the other vehicle. If the user touches two points on their own vehicle's marker and another vehicle's marker and rotates these two points while their own vehicle and the other vehicle are not following each other, a command is issued ordering the user's vehicle to overtake the other vehicle.

[0057] The specific control method for a command can be modified depending on the context. For example, while moving the user's own vehicle marker in front of a vehicle ahead (another vehicle marker) will generally issue a command ordering the user's own vehicle to overtake the vehicle ahead, a combination of methods can be used where the method of overtaking changes depending on the context. For instance, although Japanese traffic laws generally permit overtaking from the right lane, the user's own vehicle can overtake from the left lane if no other vehicle is present in the left lane. Furthermore, the overtaking maneuver can be performed along a route that reflects the driver's preference based on personal preference data.In the case of a specific control procedure associated with a command, the determination unit 11 of the HMI control device 10 can determine the specific control procedure, for example, taking into account a direction of travel, an execution time, a speed, and the like, and send a command containing the specific control procedure to the automatic driving control device 20. Alternatively, the automatic driving control device 20 can determine a specific control procedure after receiving a command.Furthermore, it is possible for the driver to select a specific control procedure by displaying the option to select a specific control procedure, determined by the HMI control device 10 or the automatic driving control device 20, on the display unit 31, or by visually / tactilely notifying the driver of a specific control procedure using a notification unit (not shown) that includes a loudspeaker or a vibration unit.

[0058] The ones described above Fig. Table 11a, illustrating automatic driving control commands, contains combinations where the content of a command changes depending on the context, even though the number of touch points, the starting position of the touch, the ending position, and the type of gesture remain the same. Context refers to information about the vehicle's surroundings, such as the relative position between the vehicle and another vehicle, the presence or absence of a lane, the presence or absence of an intersection, whether the vehicle and the other vehicle are traveling in a line or in the same lane, the vehicle's direction of travel, and similar factors.It is also possible to execute a command uniquely based solely on the number of touch points, the starting position of the touch, the ending position of the touch, and the type of gesture, without reference to the context. For example, it is possible to execute a command globally, regardless of the vehicle's surroundings, that orders a speed change when a user flicks or swipes the vehicle's marker.

[0059] Below are several examples of gesture controls used in the Fig. The 6 illustrated automatic driving control command table is described below. The following examples assume the use of a touch panel display in which the display unit 31 and the input unit 32 are combined.

[0060] Fig. Figure 7 is a flowchart illustrating an example of a gesture operation that drags and drops a vehicle icon onto a different lane. The determination unit 11 of the HMI control device 10 receives a touch event (DOWN) (S20) from the touch panel. The touch event (DOWN) is an event representing a change in the touch state of the touch panel from a non-touch state to a touch state by a finger or stylus. The determination unit 11 determines whether the coordinate detected by the touch event (DOWN) is within the display area of ​​the vehicle icon or not (S21). If the coordinate detected by the touch event is outside the display area of ​​the vehicle icon (N in S21), it is determined that the gesture operation is not a lane change command, and the process is terminated.

[0061] If the coordinate detected by the touch event is within the display area of ​​the vehicle icon (Y in S21), the determining unit 11 receives a touch event (MOTION) (S22) from the touch panel. The touch event (MOTION) is an event that represents a change from a touch state at one specific point on the touch panel to a touch state at another point on the touch panel by a finger or stylus. Then, the determining unit 11 receives a touch event (UP) (S23) from the touch panel. The touch event (UP) is an event that represents a change in the touch state on the touch panel from a touch state to a non-touch state by a finger or stylus.

[0062] The determination unit 11 determines whether the coordinate detected by the touch event (UPWARDS) is located in a different lane in the same direction as the vehicle's direction of travel (S24). If the coordinate detected by the touch event (UPWARDS) is located in a different lane in the same direction as the vehicle's direction of travel (Y in S24), the command unit 13 issues a lane-change command to the automatic driving control device 20 (S25). If the coordinate detected by the touch event (UPWARDS) is not located in a different lane in the same direction as the vehicle's direction of travel (N in S24), it is determined that the gesture operation is not a lane-change command, and the process is terminated.

[0063] Fig. Figure 8 is a diagram that shows an example of gesture control according to the flowchart of Fig. Figure 7 illustrates this. In the schematic diagram shown in Fig. As illustrated in Figure 8(a), a first lane L1 and a second lane L2 are shown, and a vehicle icon V1 is displayed on the first lane L1. Various representations of the vehicle and its surroundings, including a road, are conceivable. An actual photograph can be used, as well as an accurate CG image or an animated image. The vehicle is not limited to an icon; it can be represented as a simple marker, as text, or as an actual photograph. In other words, there are no problems as long as the vehicle is displayed as an object on a screen in some form.

[0064] If the driver wants to change the lane of their own vehicle from the first lane L1 to the second lane L2, the driver drags the vehicle icon V1 on the first lane L1 as shown in Fig. 8(a) illustrates and places the own vehicle icon V1 as in Fig. Figure 8(b) illustrates this on the second lane L2. Accordingly, a command ordering a lane change is issued.

[0065] Fig. Figure 9 is a flowchart illustrating an example of a gesture control that flicks a vehicle icon. The determination unit 11 of the HMI control device 10 receives a touch event (DOWN) (S20) from a touch panel. The determination unit 11 determines whether the coordinate detected by the touch event (DOWN) is within the display area of ​​the vehicle icon or not (S21). If the coordinate detected by the touch event is outside the display area of ​​the vehicle icon (N in S21), it is determined that the gesture control is not a lane change command, and the process is terminated.

[0066] If the coordinate detected by the touch event is within the display area of ​​the vehicle icon (Y in S21), the detection unit 11 determines whether there are multiple drivable lanes in the vehicle's direction of travel (S212). If multiple drivable lanes are present (Y in S212), the image generation unit 12 creates a drop-down area to command a lane change to the drivable lane to the left and / or right of the vehicle icon and displays the drop-down area on the touch panel (S213). If there are no multiple drivable lanes (N in S212), the process of S213 is skipped.

[0067] The determination unit 11 determines whether or not an intersection exists within a predetermined distance in the direction of travel (S214). If an intersection exists within the predetermined distance in the direction of travel (Y in S214), the image generation unit 12 creates a drop-down area to the left and / or right of the vehicle icon or the drop-down area for commanding a lane change, in order to command a course change in a direction where a course change is possible at the intersection, and displays the drop-down area for commanding the course change on the touch panel (S215). If no intersection exists within the predetermined distance in the direction of travel (N in S214), the process of S215 is skipped.

[0068] The determination unit 11 receives a touch event (UP) from the touch panel (S216). The determination unit 11 determines whether the coordinate detected by the touch event (UP) is within the storage area or not (S217). If the coordinate detected by the touch event (UP) is not within the storage area (N in S217), it is determined that the gesture operation is not a lane change command, and the process is terminated. If the coordinate detected by the touch event (UP) is within the storage area for commanding a lane change (the storage area for commanding a lane change in S217), the command unit 13 issues a lane change command to the automatic driving control device 20 (S218).If the coordinate detected by the touch event (UPWARDS) is located in the course change command storage area (the course change command storage area in S217), the command unit issues a course change command to the automatic flight control device 20 (S219).

[0069] Fig. 10A and Fig. 10B are diagrams that provide an example of gesture operation according to the flowchart from Fig. 9 illustrates this. If the driver wants to change the lane of their own vehicle from the first lane L1 to the second lane L2, the driver touches as shown in Fig. 10A(a) illustrates a vehicle icon V1 on the first lane L1. When the vehicle icon V1 is touched, the following occurs as shown in Fig. Figure 10B(b) illustrates the drop area A1 for commanding a lane change to the second lane L2. When the user flicks the vehicle icon V1 onto drop area A1, a lane change command is issued to command the lane change to the second lane L2.

[0070] If, as in Fig. 10B(a) illustrates that further ahead an intersection is located as in Fig. Figure 10B(b) illustrates a drop zone A1 for commanding a lane change to the right, a drop zone A2 for commanding a lane change to the left, a drop zone A3 for commanding a right turn, and a drop zone A4 for commanding a left turn. The user flicks the vehicle icon V1 onto any of the drop zones, thereby issuing the corresponding command.

[0071] Fig. Figure 11 is a flowchart illustrating an example of a gesture operation that touches two points on a vehicle icon and a third-party vehicle icon and rotates both points. The determination unit 11 of the HMI control device 10 receives a two-point touch event (DOWN) from the touch panel (S38). The determination unit 11 determines whether the coordinate of one point detected by the touch event (DOWN) is within the display area of ​​the vehicle icon or not (S39). If the coordinate of one point detected by the touch event (DOWN) is outside the display area of ​​the vehicle icon (N in S39), it is determined that the gesture operation is not an overtaking command, and the process is terminated.

[0072] If the coordinate of one point detected by the touch event (DOWN) is within the display area of ​​the vehicle's own icon (Y in S39), the determination unit 11 determines whether the coordinate of the other point detected by the touch event (DOWN) is within the display area of ​​the other vehicle icon (the icon of the vehicle ahead) or not (S310). If the coordinate of the other point detected by the touch event (DOWN) is outside the display area of ​​the other vehicle icon (the icon of the vehicle ahead) (N in S310), it is determined that the gesture operation is not an overtaking command, and the process is terminated.

[0073] If the coordinate of the other point detected by the touch event (DOWN) is located within the display area of ​​the other vehicle icon (the icon of the vehicle ahead) (Y in S310), the determination unit 11 receives a touch event (MOTION) from the touch panel (S311). Then, the determination unit 11 receives a two-point touch event (UP) from the touch panel (S312). The determination unit 11 determines whether a coordinate detected by the touch event (UP) on the own vehicle icon is located before the coordinate detected by the touch event (UP) on the other vehicle icon (S313). When the own vehicle icon is in front of the foreign vehicle icon (Y in S313), the command unit 13 issues an overtaking command to the automatic driving control device 20 to overtake the other vehicle corresponding to the foreign vehicle icon (the icon of the vehicle in front) (S314).If the own vehicle icon is not in front of the other vehicle icon (N in S313), it is determined that the gesture operation is not a command to overtake, and the process is terminated.

[0074] Fig. 12 is a diagram that shows an example of gesture control according to the flowchart of Fig. 11 illustrates this. If the driver wants to overtake another vehicle, the driver touches as shown in the diagram. Fig. 12(a) illustrates using two fingers to display a user's own vehicle icon (V1) and a foreign vehicle icon (V2). The user then changes the order of the user's own vehicle icon (V1) and the foreign vehicle icon (V2) and places them down. A command to overtake is then issued. In addition to the above operation, a gesture control that changes the relative positions of the user's own vehicle icon and the foreign vehicle icon can be used to issue an overtaking command.

[0075] Fig. Figure 13 is a flowchart illustrating an example of a gesture operation that drags and drops a user's own vehicle icon onto a third-party vehicle icon. The destination unit 11 of the HMI control unit 10 receives a touch event (DOWN) from the touch panel (S40). The destination unit 11 determines whether the coordinate detected by the touch event (DOWN) is within the display area of ​​the user's own vehicle icon or not (S41). If the coordinate detected by the touch event (DOWN) is outside the display area of ​​the user's own vehicle icon (N in S41), it is determined that the gesture operation is not a command to follow, and the process is terminated.

[0076] If the coordinate detected by the touch event (DOWN) is within the display area of ​​the vehicle icon (Y in S41), the determination unit 11 receives a touch event (MOTION) from the touch panel (S42). Then the determination unit 11 receives a touch event (UP) from the touch panel (S43).

[0077] The detection unit 11 determines whether the coordinate detected by the contact event (UPWARDS) is located within the display area of ​​the other vehicle icon (S44). If the coordinate detected by the contact event (UPWARDS) is located within the display area of ​​the other vehicle icon (Y in S44), the command unit 13 issues a follow-follow command to the automatic driving control device 20 (S45), thereby ordering the own vehicle to follow the other vehicle corresponding to this other vehicle icon. The image generation unit 12 displays a follow-follow icon in the schematic diagram, indicating that the own vehicle is following the other vehicle, at a position between the own vehicle icon and the other vehicle icon (S46).If the coordinate detected by the touch event (UP) is outside the display area of ​​the foreign vehicle icon (N in S44), it is determined that the gesture control is not a command to follow, and the process is terminated.

[0078] Fig. 14 is a diagram that shows an example of gesture control according to the flowchart of Fig. Figure 13 illustrates this. If the driver wants to follow another vehicle with their own vehicle, the driver pulls as shown in Figure 13. Fig. 14(a) illustrates a custom vehicle icon V1, and places the custom vehicle icon V1 as in Fig. 14(b) illustrates this on a third-party vehicle icon V2. Accordingly, a command to follow is issued, and as in Fig. Figure 14(c) shows a following icon F1, indicating that the user's vehicle is following the other vehicle, displayed in a position between the user's vehicle icon V1 and the other vehicle icon V2. The following icon F1 can be an icon indicating a coupled state between the user's vehicle and the other vehicle, such as a feather, a band, or an arrow pointing at both vehicles, or it can be a written description such as "Following". The following icon F1 can be displayed in a position other than between the user's vehicle icon and the other vehicle icon. In addition to the above operation, another operation linking the user's vehicle icon and the other vehicle icon can be used as a gesture control to issue the following command.

[0079] Fig. Figure 15 is a flowchart illustrating an example of gesture control. During a following maneuver, the gesture touches two points on a vehicle icon and a vehicle icon, increasing / decreasing the distance between the vehicle icons. The HMI control unit 10's destination unit 11 receives a two-point touch event (DOWN) from the touch panel (S412). The destination unit 11 determines whether the coordinate detected by the first touch event (DOWN) is within the display area of ​​the vehicle icon (S413). If the coordinate detected by the first touch event is outside the display area of ​​the vehicle icon (N in S413), the gesture control is determined not to be a command to change the vehicle distance, and the process terminates.

[0080] If the coordinate detected by the first touch event (DOWN) is within the display area of ​​the own vehicle icon (Y in S413), the determination unit 11 determines whether the coordinate detected by the second touch event (DOWN) is within the display area of ​​the other vehicle icon or not (S414). If the coordinate detected by the second touch event is outside the display area of ​​the other vehicle icon (N in S414), it is determined that the gesture operation is not a command to change the vehicle distance, and the process is terminated.

[0081] If the coordinate detected by the second touch event (DOWN) is within the display area of ​​the foreign vehicle icon (Y in S414), the detection unit 11 receives a touch event (MOTION) from the touch panel (S415). Then, the detection unit 11 receives a touch event (UP) from the touch panel (S416). The detection unit 11 compares the distance between a third and a fourth coordinate detected by the touch event (UP) with the distance between the first and second coordinates detected by the touch event (DOWN) (S417). If the two distances differ (Y in S417), the command unit issues a command to change the following distance to the automatic driving control device 20 (S418), in order to change the distance between the own vehicle and the other vehicle driving one behind the other.Image generation unit 13 changes the vehicle distance between the own vehicle icon and the other vehicle icon in the schematic diagram (S419). When the two distances are equal (N in S417), it is determined that the gesture operation is not a command to change the vehicle distance, and the process is terminated.

[0082] Fig. 16 is a diagram that shows an example of gesture control according to the flowchart of Fig. Figure 15 illustrates this. If the driver wishes to change the distance between their own vehicle and another vehicle while following, the driver touches the other vehicle as shown in Figure 15. Fig. 16(a) illustrates a vehicle icon V1 and a vehicle icon V2 using two fingers. If the driver is in this state as in Fig. 16(b) illustrates the vehicle spacing; when a command is issued to change the vehicle spacing in order to reduce the distance between the user's own vehicle and the other vehicle, the vehicle spacing after the change is determined according to the distance between the user's own vehicle icon V1 and the other vehicle icon V2. Fig. Figure 16(c) illustrates a schematic diagram after the change in vehicle spacing, where the vehicle spacing between the own vehicle icon V1 and the other vehicle icon V2 is compared with the vehicle spacing in Fig. 16(a) is less. If the driver, while touching the own vehicle icon V1 and the other vehicle icon V2, pulls up the vehicle distance between the own vehicle icon V1 and the other vehicle icon V2 with two fingers, a command to change the vehicle distance is issued to increase the distance between the own vehicle and the other vehicle. In addition to the two-finger pull-up / pulling operation, a one-finger operation can also be used as a gesture control to issue the command to change the vehicle distance, provided that an operation to change the distance between the own vehicle icon and the other vehicle icon is used.

[0083] Fig. Figure 17 is a flowchart illustrating an example of gesture control where, while following another vehicle, the user touches two points on a vehicle icon (one of their own) and a vehicle icon (the other) and rotates the two points. The processes from step S412 to step S416 are the same as those in the flowchart of [reference missing]. Fig. 15. The determination unit 11 determines whether the midpoint between a third and a fourth coordinate detected by the contact event (UP) is the same as the midpoint between a first and a second coordinate detected by the contact event (DOWN) (S427). If the midpoint between the third and fourth coordinates is the same as the midpoint between the first and second coordinates (Y in S427), the command unit 13 issues a command to the automatic driving control device 20 to cancel the following action (S428). The command unit 13 sends a vehicle-to-vehicle follow command to the other vehicle via vehicle-to-vehicle communication, instructing the other vehicle to follow the first vehicle (S429).The command to follow another vehicle can be sent to the other vehicle via a relay device. The image generation unit 12 swaps the own vehicle icon (the icon of the following vehicle) and the other vehicle icon (the icon of the vehicle being followed) in the schematic diagram (S430). If the positions of the two center points differ in step S427 (N in S427), it is determined that the gesture control is not a command to swap the following vehicle, and the process is terminated.

[0084] Fig. 18 is a diagram that shows an example of gesture control according to the flowchart of Fig. 17 illustrates this. If the driver wishes to exchange the positions of his own vehicle and the other vehicle while following, the driver touches as shown in Fig. 18(a) illustrates a self-operated vehicle icon V1 and a foreign vehicle icon V2 with two fingers. In this state, the driver turns as in Fig. Figure 18(b) illustrates the two fingers in such a way that the own vehicle icon V1 and the other vehicle icon V2 are swapped. Accordingly, a command is issued to swap the positions of the own vehicle and the other vehicle when following. Fig. 18(c) illustrates a schematic diagram after the vehicles have been swapped and compares the positions of the own vehicle icon V1 and the foreign vehicle icon V2 with the positions in Fig. 18(a) conversely. In Fig. 18(a) and Fig. In diagram 18(B), an imaginary line L10 and a midpoint C2 between the own vehicle icon V1 and the other vehicle icon V2 are shown in the schematic diagram. However, an example can also be used in which the imaginary line L10 and the midpoint C2 are not shown. The following order is periodically reversed, making it possible to equalize the fuel consumption of the own vehicle and the other vehicle. In addition to the above operation, another operation that disconnects the own vehicle icon and the other vehicle icon can be used to issue a command to stop following. As a gesture control that issues a command to change the following relationship, in addition to the above operation, an operation that reverses the following relationship between the own vehicle icon and the other vehicle icon can also be used.The own vehicle icon and the other vehicle icon can be swapped by displaying a swap button within the own vehicle icon or the other vehicle icon, or between the own vehicle icon and the other vehicle icon, and pressing the swap button. After a following relationship has been established, the automatic driving control device 20 can periodically perform a swapping operation.

[0085] Fig. Figure 19 is a flowchart illustrating an example of a gesture control that swipes a vehicle icon. The determination unit 11 of the HMI control device 10 receives a touch event (DOWN) from the touch panel (S431). The determination unit 11 determines whether the coordinate detected by the touch event (DOWN) is within the display area of ​​the vehicle icon or not (S432). If the coordinate detected by the touch event (DOWN) is outside the display area of ​​the vehicle icon (N in S432), it is determined that the gesture control is not a command to set the vehicle distance, and the process is terminated.

[0086] If the coordinate detected by the touch event (DOWN) is within the display area of ​​the own vehicle icon (Y in S432), the detection unit 11 receives a touch event (MOTION) from the touch panel (S433). The image generation unit 12 displays the vehicle distance between the own vehicle and the other vehicle in the schematic diagram (S434). During a period in which no touch event (UP) is received from the touch panel (N in S435), the process proceeds to step S433, and the receipt of a touch event (MOTION) and the display of the vehicle distance information continue (S433 and S434). That is, while the distance between the own vehicle icon and the other vehicle icon changes, the image generation unit 12 updates the vehicle distance information in the schematic diagram in real time.

[0087] When a touch event (UP) is received from the touch panel (Y in S435), the detection unit 11 determines whether the coordinate detected by the touch event (UP) is on the same lane as the own vehicle icon and behind the other vehicle icon (S436). If the coordinate detected by the touch event (UP) is on the same lane as the own vehicle icon and behind the other vehicle icon (Y in S436), the command unit issues a command to the automatic driving control device 20 to set the vehicle distance according to the distance between the own vehicle icon and the other vehicle icon (S437). The image generation unit 12 displays the set vehicle distance information in the schematic diagram (S438).If the coordinate detected by the touch event (UPWARDS) is not on the same lane as the own vehicle icon and is behind the other vehicle icon (N in S436), it is determined that the gesture operation is not a command to set the vehicle distance, and the process is terminated.

[0088] Fig. 20 is a diagram that shows an example of gesture control according to the flowchart of Fig. Figure 19 illustrates this. If the driver wants to set a vehicle distance between their vehicle and another vehicle, the driver wipes as shown. Fig. 20(a) illustrates a custom vehicle icon V1 as in Fig. 20(b) illustrates swiping up or down. During the swipe, a pop-up window P1 is displayed to show the vehicle distance. As in Fig. Figure 20(c) illustrates that upon completion of the swipe, a command to set the vehicle distance is issued. A message indicating that a vehicle distance has been set is displayed in the popup window P1, after which the popup window P1 is cleared.

[0089] Fig. Figure 21 is a flowchart illustrating an example of dragging and dropping a vehicle icon onto a specific area (a road sign). The HMI control unit 10's determination unit 11 receives a touch event (DOWN) from the touch panel (S50). Determination unit 11 then determines whether the coordinate detected by the touch event (DOWN) is within the display area of ​​the vehicle icon (S51). If the coordinate detected by the touch event (DOWN) is outside the display area of ​​the vehicle icon (N in S51), it is determined that the gesture operation is not a command corresponding to the content of the road sign, and the process is terminated.

[0090] If the coordinate detected by the touch event (DOWN) is within the display area of ​​the vehicle's own area (Y in S51), the determining unit 11 receives a touch event (MOTION) from the touch panel (S52). Then the determining unit 11 receives a touch event (UP) from the touch panel (S53).

[0091] The determination unit 11 determines whether the coordinate detected by the touch event (UP) is within a display area of ​​the road sign or not (S54). If the coordinate detected by the touch event (UP) is within the display area of ​​the road sign (Y in S54), the command unit 13 issues a command corresponding to the content of the road sign to the automatic driving control device 20 (S55). If the coordinate detected by the touch event (UP) is outside the display area of ​​the road sign (N in S54), it is determined that the gesture operation is not an operating command corresponding to the content of the road sign, and the process is terminated.

[0092] Fig. 22 is a diagram that shows an example of gesture control according to the flowchart of Fig. 21 illustrates this. If the driver wishes to change the course of their vehicle to one of the courses shown on a destination signpost S1 at an intersection, the driver pulls as shown in Fig. 21(a) illustrates a custom vehicle icon V1 and places the custom vehicle icon V1 as in Fig. 22(b) illustrates the location name “Machida” N1 of the desired destination on the destination signpost S1. Accordingly, a route change command is issued to change the shortcut of the own vehicle to a route on which the desired destination is specified as a transit point or as a destination.

[0093] The following are under Return to Fig. 5. Details of the processing of drawing a predictive trajectory / route candidate are described in step S9e.

[0094] Fig. 23 is a flowchart that provides an example of the processing of drawing a predictive trajectory / route candidate in step S9e of Fig. Figure 5 illustrates this. The determination unit 11 of the HMI control device 10 determines whether or not a movement signal of the vehicle icon is received from the touch panel (S9ea). If a movement signal of the vehicle icon is received (Y in S9ea), the determination unit 11 determines whether the vehicle icon's movement position is on a lane or not (S9eb). If the vehicle icon's movement position is on a lane (Y in S9eb), the determination unit 11 determines whether the vehicle icon's movement position is on a different lane than the lane on which the vehicle icon's movement began (S9ec).If the movement position of the vehicle icon is not on a different lane than the lane on which the movement of the vehicle icon began (if the movement position of the vehicle icon is on the same lane as the lane on which the movement of the vehicle icon began) (N in S9ec), the image generation unit 12 deletes a prediction path in the schematic diagram displayed on the touch panel (S9ed). In step S9eb, the image generation unit 12 also deletes a prediction path in the schematic diagram displayed on the touch panel if the movement position of the vehicle icon is not on a lane (N in S9eb).

[0095] In step S9ec, the determination unit 11 determines, if the movement position of the vehicle icon is in a different lane than the lane in which the movement of the vehicle icon began (Y in S9ec), whether the movement position of the vehicle icon is in a lane to which a lane change is possible or not (S9ee). If the movement position of the vehicle icon is not in a lane to which a lane change is possible (N in S9ee), the image generation unit 12 records an error message on the touch panel (S9ef). If the movement position of the vehicle icon is in a lane to which a lane change is possible (Y in S9ee), the command unit 13 reports the coordinate of the touch event on the vehicle icon to the automatic driving control device 20 and requests a predicted path (S9eg).The image generation unit 12 plots the predicted path of the lane change, received from the automatic driving control device 20, onto the touch panel (S9eh). The above processing continues until the touch panel no longer receives a movement signal from the vehicle's own icon (N in S9ea).

[0096] Fig. 24 is a flowchart that shows another example of the processing of drawing a predictive trajectory / route candidate in step S9e of Fig. Figure 5 illustrates the processes from step S9ea to step S9ef. These are the same as those in the flowchart of [reference missing]. Fig. 23. If the movement position of the vehicle icon is on a lane to which a lane change is possible (Y in S9ee), the determination unit 11 determines whether or not there is another vehicle on the lane representing the target of the lane change (S9ei). If there is no other vehicle on the lane representing the target of the lane change (N in S9ei), the image generation unit 12 draws a predictive path of the lane change onto the touch panel without acceleration or deceleration (S9ej).

[0097] If another vehicle is present in the lane representing the target of the lane change (Y in S9ei), the determination unit 11 determines whether the movement position of the own vehicle icon is in front of or behind the other vehicle icon (S9ek). If the movement position of the own vehicle icon is in front of the other vehicle icon ("in front" in S9ek), the image generation unit 12 draws a predictive lane change path with acceleration on the touch panel (S9e1). If the movement position of the own vehicle icon is behind the other vehicle icon ("behind" in S9ek), the image generation unit 12 draws a predictive lane change path with deceleration on the touch panel (S9em). If the movement position of the own vehicle icon is within the other vehicle icon ("other vehicle" in S9e), the image generation unit 12 draws an error message on the touch panel (S9en).The above processing continues until the touch panel no longer receives a movement notification of the vehicle icon (N in S9ea).

[0098] Fig. 25A to Fig. 25C are diagrams that provide an example of gesture control according to the flowchart of Fig. 24 illustrate. Fig. 25A is an example of a predicted path of a lane change without acceleration and deceleration in step S9ej. Fig. 25B is an example of a predicted trajectory of a lane change with acceleration in step S9el. Fig. 25C is an example of a predicted trajectory of a lane change with deceleration at step S9em.

[0099] Fig. Figure 26 is a diagram illustrating an example of an error screen display. If the driver wants to change their vehicle's lane from the first lane L1 to the second lane L2, the driver pulls the lever as shown in Figure 26. Fig. 26(a) illustrates the own vehicle icon V1 on the first lane L1 and places the own vehicle icon V2 as in Fig. 26(b) illustrates this on the second lane V2. When another vehicle is detected approaching the position of the own vehicle icon, as shown in Fig. 26(c) illustrates in the popup window P1 an error message “The lane change is being reversed due to a following vehicle” is displayed and the own vehicle icon V1 is returned to the first lane L1.

[0100] Fig. 27 is a diagram illustrating a display example during a period in which the driver's own vehicle icon V1 is dragged and dropped onto another vehicle icon V2. If the driver wants to follow another vehicle, the driver drags as shown in Fig. 27(a) illustrates the own vehicle icon V1. During the period in which the own vehicle icon V1 is dragged onto and dropped onto the other vehicle icon V2, as shown in Fig. Figure 27(b) illustrates that an afterimage V1a (a dashed circle marking or the like) is shown at the original position of the own vehicle icon. As in Fig. 27(c) illustrates that the afterimage is deleted when the own vehicle icon V1 is placed on the other vehicle icon V2. As shown in Fig. 27(d) illustrates that after establishing following, the following icon F1 is displayed between the own vehicle icon V1 and the other vehicle icon V2.

[0101] Fig. Figure 28 is a diagram illustrating another display example during a period in which the user's own vehicle icon V1 is dragged and dropped onto a third-party vehicle icon V2. Fig. 28(a) illustrates an example in which, during the period in which the own vehicle icon V1 is dragged onto and dropped onto the other vehicle icon V2, drop-off areas are displayed in the vicinity of a road area. Fig. 28(a) shows a return drop-off area D1, a drop-off area for home rerouting D2, a left-turn drop-off area D3, and a right-turn drop-off area D4, each located above, below, to the right, and to the left of the road area, respectively. The driver places the vehicle icon V1 on any of the drop-off areas D11 to D14, thereby enabling the corresponding operation. The drop-off area may display a different operation than the one described above, and may, for example, display an operating command such as rerouting to a gas station, rerouting to a parking area, rerouting to a rest area, overtaking, changing lanes, adaptive cruise control (ACC), accelerating and decelerating to a target speed, or the like.

[0102] In Fig. 28(b) illustrates an example in which, during the period in which the own vehicle icon V1 is dragged and dropped onto a foreign vehicle icon V2, the color of a background such as a road or the like is changed (inverted, lightened, or the like), and the background color is reset to its original color after the dropping process is complete. Fig. 28(c) illustrates an example in which, during the period in which the own vehicle icon V1 is dragged onto and placed on the foreign vehicle icon V2, a movement path T1 (a dashed line or the like) is drawn from the starting position of the drag and the path T1 is deleted after the placement is completed.

[0103] Fig. 29 is a diagram illustrating an example of how toggled a self-vehicle icon V1 when there is an area where dropping is not possible, or similar. If the driver wants to drive their own vehicle behind the other vehicle, the driver drags it as shown in Fig. 29(a) illustrates the own vehicle icon V1. After the own vehicle icon V1 has been dragged, its color is changed (inverted, lightened, or the like) if there is an area where dropping is not possible (a vehicle refusing to follow, an oncoming lane, or the like), or if the operation to follow is not possible depending on the circumstances. If dropping onto the other vehicle icon V2 is not possible, in the case of a situation as described in Fig. 29(b) illustrated placing the own vehicle icon V1 onto the other vehicle icon V2 the own vehicle icon V1 as in Fig. 29(c) illustrates that the vehicle has been returned to its original position and a popup window P3 is shown displaying an error message such as "this is an area where following operation is not possible".

[0104] Fig. 30A to Fig. 30C are diagrams that illustrate another display example for a drawn own vehicle icon V1 when there is an area where placement is not possible, or something similar. Fig. Example 30A illustrates how, after dragging the vehicle icon V1, the color of a background, such as a road or similar, changes if following the vehicle is not possible. If following the vehicle is possible, the background color is reset to its original color. Fig. Figure 30B illustrates an example where, after dragging the custom vehicle icon V1, the color of an area where dropping is not possible is changed (inverted, lightened, or similar). Fig. 30C illustrates an example where, after dragging the vehicle icon V1, a message using an error signal or vibration is displayed if there is an area where dropping is not possible or if operation to follow is not possible.

[0105] Fig. Figure 31 is a diagram illustrating an example of how a pull operation is not possible. If the driver wants to follow another vehicle, the procedure is as shown in... Fig. 31(a) illustrates at the beginning of the process of drawing the own vehicle icon V1 as in Fig. Figure 31(b) illustrates that the color of the vehicle icon V1 has been changed and that dragging (moving the icon) is made impossible when operation cannot be initiated. Additionally, a popup window P3 is shown displaying an error message such as "this is an area where following operation is not possible".

[0106] Fig. 32A and Fig. 32B are diagrams that illustrate an alternative display example when a drag operation is not possible. In Fig. 32(a) illustrates an example where, at the beginning of dragging the vehicle icon V1, the color of a background, such as a road or the like, is changed if it is not possible to initiate a follow-up operation. If a follow-up operation is possible, the background color is reset to its original color. Fig. 32(b) illustrates an example where, at the beginning of the dragging of the own vehicle icon V1 in a state where no dragging operation can be performed, a message using an error signal or vibration is given if it is not possible to start an operation.

[0107] Fig. Figure 33 is a diagram illustrating a first example of how to control the vehicle icon during the process of placing the icon and completing an overtaking maneuver. If the driver wants to overtake a vehicle ahead, the driver pulls the lever as shown in Figure 33. Fig. 33(a) illustrates the own vehicle V1 and places the own vehicle icon V1 as in Fig. 33(b) illustrates this in front of a foreign vehicle icon V2. During the period in which the own vehicle icon V2 is placed in front of the foreign vehicle icon V2 and the own vehicle completes the overtaking maneuver, as in Fig. Figure 33(c) illustrates the state (current position) of the own vehicle, represented as a ghost icon V1g. The path T1 of the movement is also shown. When the overtaking maneuver is complete, the ghost icon V1g and the path T1 of the own vehicle are displayed as shown in Fig. 33(d) illustrated deleted.

[0108] Fig. Figure 34 is a diagram illustrating a second example of how to control the vehicle icon during the process of placing the vehicle icon and completing an overtaking maneuver. If the driver wishes to overtake a vehicle ahead, the driver pulls the lever as shown in Figure 34. Fig. 34(a) illustrates the own vehicle V1 and places the own vehicle icon V1 as in Fig. 34(b) illustrates this in front of a foreign vehicle icon V2. During the period in which the own vehicle icon V2 is placed in front of the foreign vehicle icon V2 and the own vehicle completes the overtaking maneuver, as in Fig. Figure 34(c) illustrates how the display state of the vehicle icon V1, which represents the destination of the movement, changes. Fig. 34(c) illustrates an example where the vehicle V1 flashes its lights, but a change in color, size, position, or the like is also possible. When the overtaking maneuver is completed, the following occurs as shown in Fig. Figure 34(d) illustrates the display state of the vehicle icon V1 being reset to its original display state.

[0109] Fig. Figure 35 is a diagram illustrating a third example of how to control the vehicle icon during the process of placing the vehicle icon and completing an overtaking maneuver. If the driver wishes to overtake a vehicle ahead, the driver pulls the lever as shown in Figure 35. Fig. 35(a) illustrates the own vehicle V1 and places the own vehicle icon V1 as in Fig. 35(b) illustrates this in front of a foreign vehicle icon V2. During the period in which the own vehicle icon V2 is placed in front of the foreign vehicle icon V2 and the own vehicle completes the overtaking maneuver, an additional operation queues up a next command (reserves a control to be performed after the current control). In Fig. 35(c) to Fig. Paragraph 35(d) illustrates an example where a lane change is also commanded during the control of the overtaking maneuver. Despite the overtaking maneuver being controlled as follows: lane change to an overtaking lane → overtaking the other vehicle → lane change back to the original lane → lane change to an overtaking lane, in this case the last two control processes are cancelled and the control of the overtaking maneuver is therefore as follows: lane change to an overtaking lane → overtaking the other vehicle.

[0110] Fig. Figure 36 is a diagram illustrating a third example of how to control the vehicle icon during the process of placing the vehicle icon and completing an overtaking maneuver. If the driver wishes to overtake a vehicle ahead, the driver pulls the lever as shown in Figure 36. Fig. 36(a) illustrates the own vehicle V1 and places the own vehicle icon V1 as in Fig. 36(b) illustrates this in front of a foreign vehicle icon V2. During the period in which the own vehicle icon V2 is placed in front of the foreign vehicle icon V2 and the own vehicle completes the overtaking maneuver, as in Fig. 36(c) and Fig.36(d) illustrates that when the vehicle icon V1 is dragged again and placed back in its original position, the following processing takes place. When the vehicle icon V1 is placed back in its original position, a command ordering a lane change to the original lane is transmitted when a lane change to an overtaking lane is initiated or completed. When the vehicle icon V1 is placed back in its original position, an error message (for example, "The operation cannot be performed because a control operation is taking place," "Please wait a moment," or similar) is displayed when the vehicle icon V1 is placed back in its original position.

[0111] As described above, the present embodiment allows the contents of various controls to be transferred to the automatic driving control device 20 by moving an icon displayed on the touch panel using gesture control. Gesture control of the icon is a simple operation, thus relieving the driver of conventional driving controls such as turning the steering wheel 51, pressing the accelerator pedal 53, or the like. The driver can simultaneously check the ambient conditions and issue an operating command on the touch panel, meaning the driver's line of sight remains unobstructed. This reduces the likelihood of incorrect operation and promotes safer driving.

[0112] The present invention has been described above based on the embodiments. These embodiments are shown only by way of example, and those skilled in the art should understand that the embodiments can be modified by combining the respective elements or processing methods in various ways, and that these examples of modification are included in the scope of the present invention.

[0113] Although, for example, an example has been assumed in which the HMI control device 10 is implemented by a dedicated LSI, the functions of the HMI control device can be implemented using a central processing unit (CPU) in a portable device such as a smartphone or tablet, which is used as a display device 30. In this case, the portable device used as a display device 30 and the automatic driving control device 20 are directly connected. The functions of the HMI control device 10 can be implemented by a CPU in a head unit such as a vehicle navigation system, a display audio system, or the like. The head unit can contain a dedicated LSI to which the HMI control device 10 is attached.

[0114] The embodiments described above provided some examples of gesture control where a vehicle icon is defined as the starting point of the touch gesture. Examples of gesture control where the vehicle icon is defined as the starting point of the touch gesture, including the embodiments described above, are given below.

[0115] A command to increase speed is issued via a gesture control: the user touches the vehicle icon and swipes it forward, provided there is no obstacle in the same lane. A command to decrease speed is issued via a gesture control: the user swipes the vehicle icon backward, provided there is no obstacle in the same lane. In this case, there is no need to prepare a separate user interface, thus saving space. A command to stop right or left in the same lane is issued via a gesture control: the user touches the vehicle icon and swipes it right or left.In this case, it is possible to perform the user input and display the result on the same screen, thus saving space.

[0116] A command to follow another vehicle or activate adaptive cruise control (ACC) is issued via a gesture control where the vehicle icon is dragged and dropped onto the other vehicle icon. A command to overtake is issued via a gesture control where the vehicle icon is dragged and dropped in front of the other vehicle icon. In this case, there is no need to prepare a separate user interface, thus saving space.

[0117] When multiple lanes are available, a lane change command is issued via a gesture control where the vehicle icon is dragged and dropped into another lane. This allows the input and the result to be displayed on the same screen, saving space. If another vehicle is in a different lane, the vehicle's position in front of or behind the other vehicle's icon determines whether the vehicle merges in front of or behind the other vehicle.If a right-turn lane and / or a left-turn lane is present, a command that orders a right turn or a left turn after a lane change is issued via a gesture control, in which the vehicle icon is placed on the right-turn lane or the left-turn lane. In this case, one step of entering a right-turn or left-turn command can be omitted.

[0118] A command to enter a toll road is issued via a gesture control where the vehicle icon is dragged and dropped onto an entry icon of the toll road (for example, a highway). In this case, it is possible to omit an input step that would otherwise be required for the toll road entry icon. Alternatively, a command to enter a toll road can be issued via a gesture control where the vehicle icon is dragged and dropped onto an entry lane of the toll road. In this case, the user input and the display of the result can be performed on the same screen.

[0119] A command to exit a toll road is issued via a gesture control where the vehicle icon is dragged and dropped onto an exit icon of the toll road. In this case, it is possible to omit an input step that would otherwise be required for the exit icon. Alternatively, a command to exit a toll road can be issued via a gesture control where the vehicle icon is dragged and dropped onto an exit lane of the toll road. In this case, it is possible to perform the input and display the result on the same screen.

[0120] On an acceleration lane at a highway on-ramp, a command to control the entry, specifying whether the driver's vehicle should position itself in front of or behind another vehicle according to the speed of the approaching vehicle, is issued via a gesture control. This involves dragging and dropping the driver's vehicle icon onto a lane. In this case, the input and the display of the result can be performed on the same screen.

[0121] A command to change course is issued via a gesture control, where the vehicle icon is dragged and dropped onto a lane in the desired turning direction. In this case, the input and the result are displayed on the same screen. For example, dropping the vehicle icon onto a right-turn lane will issue a command to turn right. A command to perform a U-turn is issued via a gesture control, where the vehicle icon is dragged and dropped to form the letter "U". In this case, the input and the result are displayed on the same screen.

[0122] A command to change the destination or add a transit point is issued via a gesture control where the vehicle icon is dragged and dropped onto an icon indicating a destination or point of interest. An icon indicating the user's residence can always be displayed on the screen. A command to drive in a direction in which the vehicle is flicked, without specifying a destination, is issued via a gesture control where the vehicle icon is touched and then flicked in the desired direction of travel.

[0123] When the driver drags and drops their vehicle icon onto a desired parking space in a parking lot (the space is defined in an approximate area such as the vicinity of a store entrance or similar), a command is issued ordering the vehicle to park in an empty space within that space.

[0124] If the driver drags and drops their vehicle icon onto a pedestrian icon, a mastic icon, a bus stop icon, or a taxi rank icon, a command is issued ordering the vehicle to move aside at the location where the icon was placed. In this case, it is possible to shorten the step required to issue a stop command.

[0125] A drag-and-drop interface allows you to define a "Point of Interest (POI)" area. For example, after defining a search area, you can search for a gas station within that area. A command to search for a parking space can be issued via a gesture control, dragging and dropping your vehicle icon onto a rest area or parking space icon.

[0126] A command to illuminate a pedestrian using the headlights is issued via a gesture control where the vehicle icon is dragged and dropped onto a pedestrian icon. This eliminates the step required to activate the headlights. Similarly, a command to sound the horn at a pedestrian is issued via a gesture control where the vehicle icon is dragged and dropped onto a pedestrian icon. Depending on the pedestrian's position, a directional horn can be used. This also eliminates the step required to activate the horn.

[0127] While the vehicle icon is being dragged, while the vehicle icon is being placed, or while the vehicle icon is placed and processing according to a control command is taking place, the automatic driving control device 20 receives a predictive path and can draw the predictive path on the display unit 31. If the processing corresponding to a control command is temporarily suspended because the vehicle is waiting at a traffic light, or in a section where a lane change is not possible, the image generation unit 12 displays a message such as "attempting" until the processing corresponding to the control command takes place, and can draw a predictive path at the time the processing becomes executable.While the vehicle icon is being dragged, or while the vehicle is being placed and processing corresponding to a control command is taking place, the image generation unit 12 can display an estimated time required or a remaining time required until the vehicle icon is placed and processing corresponding to the control command is taking place. If processing corresponding to a control command is temporarily suspended, the control can be preset to continue until control becomes possible, or to be suspended and stopped. Because in some cases a drag is accidentally initiated, or a retraction operation is required after a drag, a retraction drop area can be provided.If the driver places the vehicle icon on a cancellation drop area after dragging the vehicle icon, the issuance of a command can be canceled.

[0128] In the above description, if two or more types of candidates for executable actions are present, a selection menu may appear after dropping the vehicle icon. When the vehicle icon is dropped, instructions for all available operations may be displayed on the screen. After dropping the vehicle icon, the dropped state can be retained even after the icon is released, until a confirmation button is pressed. If the vehicle icon is accidentally dropped after dropping, a correction procedure and a verification procedure may be displayed via a pop-up screen.The touch panel can be designed to provide haptic feedback through different vibration modes when the vehicle icon is moved to a drop position after being dragged, and again when it is moved to a drop position. When the vehicle icon is dropped, the screen scale can be automatically adjusted. For example, the screen scale can be automatically adjusted to ensure that at least one other vehicle is displayed at the nearest location. This makes it easier to find another vehicle and issue commands more quickly. The color of the vehicle icon can also change depending on whether it is moved to a drop position or a drop position.If the vehicle icon is moved to a position where it cannot be placed after being dragged, the background color may differ between areas where placement is possible and areas where it is not. Although a gesture control corresponding to a control command is described as drag-and-drop or similar operation, touch-and-touch operation may also be used. Preferably, a predefined operation is used, but an operation customizable by the driver is also possible. Furthermore, to indicate the relationship between a gesture control and a control command, the display unit may show an icon or an arrow, or provide a guide display or voice prompt.

[0129] The embodiments can be specified by the following points. Point 1

[0130] A driving assistance device (10) comprising an image output unit (14a) which outputs an image containing a vehicle object representing the vehicle and an environmental state of the vehicle to a display unit (31); an operating signal input unit (14b) which receives a gesture control from a user which includes moving the vehicle object in the image displayed on the display unit (31); and a command output unit (14c) which outputs a command corresponding to the gesture control to an automatic driving control unit (20) which controls automatic driving.

[0131] In this case, it is possible to perform operations associated with automatic driving intuitively and conveniently. Point 2

[0132] The driving assistance device (10) according to point 1, wherein the command output unit (14c) when the operating signal input unit (14b) receives a gesture control from a user which moves the own vehicle object in the image displayed on the display unit (31) to the position of a non-own vehicle object which represents a predetermined target different from the own vehicle, issues a command which orders a change in the relative positional relationship between the own vehicle and the predetermined target, or a command which orders an output of a message from the own vehicle to the predetermined target, to the automatic driving control unit (20).

[0133] The predetermined destination, different from the vehicle itself, can be another vehicle, a pedestrian, an obstacle, or a specific area.

[0134] In this case, it is possible to perform an operation intuitively and conveniently that is linked to a predetermined destination different from one's own vehicle. Point 3

[0135] The driving assistance device (10) according to point 1, wherein the command output unit (14c) when the operating signal input unit (14b) receives a gesture control from a user which moves the own vehicle object in the image displayed on the display unit (31) to a position where there is no non-own vehicle object representing a predetermined destination other than the own vehicle, issues a command to move the own vehicle to a location corresponding to the end position of the gesture control to the automatic driving control unit (20).

[0136] In this case, it is possible to perform an operation that is not connected to the predetermined destination different from the vehicle itself (for example, a lane change, a course change, or the like) intuitively and conveniently. Point 4

[0137] The driving assistance device (10) according to point 1, wherein the command output unit (14c) when the operating signal input unit (14b) receives a gesture control from a user which changes a distance between the own vehicle object in the image displayed on the display unit (31) and a non-own vehicle object which represents a predetermined target different from the own vehicle, issues a command ordering a change in the distance between the own vehicle and the predetermined target to the automatic driving control unit (20).

[0138] In this case, it is possible to intuitively and conveniently operate the controls to change the distance between your own vehicle and the predetermined destination. Point 5

[0139] The driving assistance device (10) according to point 1, wherein the command output unit (14c) when the operating signal input unit (14b) receives a gesture control from a user which swaps the own vehicle object in the image displayed on the display unit (31) and a non-own vehicle object in the image which represents a predetermined destination other than the own vehicle, issues a command to the automatic driving control unit (20) which commands a change in the relative position relationship between the own vehicle and the predetermined destination.

[0140] In this case, it is possible to intuitively and conveniently operate a control to change the relative positional relationship between one's own vehicle and the predetermined destination. Point 6

[0141] The driving assistance device (10) according to point 1, wherein the command output unit (14c) then, when the operating signal input unit (14b) is in a state in which the own vehicle and the predetermined target other than the own vehicle are in a coupled state, receives a gesture control from a user which moves the own vehicle object in the image displayed on the display unit outside an imaginary line connecting the own vehicle object and a non-own vehicle object that represents the predetermined target other than the own vehicle, issues a command ordering the uncoupling to the automatic driving control unit.

[0142] In this case, it is possible to intuitively and conveniently perform an operation to disconnect the link between your own vehicle and the predetermined destination. Point 7

[0143] The driving assistance device (10) according to point 1, wherein the command output unit (14c) changes the content of the command to be issued to the automatic driving control unit (20) depending on the environmental state of the vehicle at the time of receiving the gesture control by the operating signal input unit (14b).

[0144] In this case, it is possible to issue a command appropriate to the context. Point 8

[0145] The driving assistance device (10) according to one of points 1 to 7, wherein the image output unit (14a) during the period in which the operating signal input unit (14b) cannot accept gesture control from a user, outputs to the display unit (31) an image in which a display mode of the vehicle object differs from the display mode during the period in which the operating signal input unit (14b) can accept gesture control from the user.

[0146] In this case, a user can easily see whether the gesture input is effective or ineffective during that period. Point 9

[0147] The driving assistance device (10) according to one of points 1 to 8, wherein the image output unit (14a) modifies sizes of the own vehicle object, the non-own vehicle object representing the predetermined target, and the environmental state of the own vehicle in such a way that an image containing the predetermined target, which is different from the own vehicle, is displayed on the display unit (31).

[0148] In this case, a user can see an image containing the non-own vehicle object. Point 10

[0149] A driving assistance system (10, 30) that comprising a display device (30) that displays an image; and a driving assistance device (10) that outputs an image to the display device (30), wherein the driving assistance device (10) outputs an image output unit (14a) which outputs an image containing a vehicle object representing the vehicle itself and an environmental state of the vehicle itself to the display device (30); an operating signal input unit (14b) which receives gesture control by a user, which includes moving the vehicle object in the image displayed on the display device (30); and a command output unit (14c) which outputs a command corresponding to the gesture operation to an automatic driving control unit (20) which controls the automatic driving.

[0150] In this case, it is possible to perform operations associated with automatic driving intuitively and conveniently. Point 11

[0151] A driver assistance procedure that a step of outputting an image containing a own vehicle object representing one's own vehicle and an environment state of one's own vehicle to a display unit (31); a step of obtaining gesture control by a user, which includes moving the self-propelled vehicle object in the image displayed on the display unit (31); and includes a step of issuing a command corresponding to the gesture control to an automatic driving control unit (20) which controls the automatic driving.

[0152] In this case, it is possible to perform operations associated with automatic driving intuitively and conveniently. Point 12

[0153] Driving assistance program that makes a computer, a processing of the output of an image containing a vehicle object representing a vehicle and an environment state of the vehicle to a display unit (31); a processing of the receipt of a gesture control by a user, which includes moving the own vehicle object in the image displayed on the display unit (31); and to process the output of a command corresponding to the gesture control to an automatic driving control unit (20) which controls automatic driving.

[0154] In this case, it is possible to perform operations associated with automatic driving intuitively and conveniently. Point 13

[0155] Autonomous vehicle (1) that an image output unit (14a) which outputs an image containing a vehicle object representing a vehicle and an environment state of the vehicle to a display unit (31); a control signal input unit (14b) which receives gesture control by a user, which includes moving the vehicle object in the image displayed on the display unit (31); a command output unit (14c) that outputs a command corresponding to the gesture operation; and includes an automatic driving control unit (20) that executes the issued command.

[0156] In this case, it is possible to perform operations associated with automatic driving intuitively and conveniently. COMMERCIAL APPLICABILITY

[0157] The present invention can be used in a vehicle equipped with an automatic driving mode. REFERENCE MARKS IN THE DRAWINGS 1 vehicle 10 Driving assistance device (HMI control device) 11 Unit of determination 11a Automatic transmission control command table 12 Image generation unit 13 Command Unit 14 Input / Output Unit 14a Image output unit 14b Operating signal input unit 14c command output unit 14d Vehicle Information Input Unit 20 Automatic driving control device (automatic driving control) 21 Control unit 22 storage units 23 Input / Output Unit 30 Display device 31 Display unit 32 Input unit 40 detection units 41 Position Information Acquisition Unit 42 Sensor 43 Speed ​​Information Acquisition Unit 44 Planning Information Acquisition Unit 50 driving unit 51 Steering 52 Brake pedal 53 Accelerator pedal 54 Turn signal switches

Claims

[1] A driving assistance device comprising: an image output unit that outputs an image including a vehicle object representing a vehicle and the surrounding state of the vehicle to a display unit; an operation signal input unit that, when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation can be accepted by a user, receives a gesture operation that moves the vehicle object in the image displayed on the display unit to a position of an object that is a predetermined destination other than the vehicle; and a command output unit that, when the user's gesture operation is received, outputs a first command commanding a change in a relative positional relationship between the vehicle and the predetermined destination, or a second command commanding an output of a message from the vehicle to the predetermined destination, to an automatic driving control unit that controls the automatic driving, wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first or second command is present and a first time has passed after the determination of the first or second command based on the gesture operation. [2] A driving assistance device comprising: an image output unit that outputs an image including a vehicle object representing a vehicle and an environmental state of the vehicle to a display unit; an operation signal input unit that, when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation can be accepted by a user, receives a gesture operation that exchanges the vehicle object in the image displayed on the display unit and a vehicle object that represents a predetermined destination different from the vehicle in the image; and a command output unit that, when the user's gesture operation is received, outputs a first command commanding a change in a relative positional relationship between the vehicle and the predetermined destination to an automatic driving control unit that controls the automatic driving, wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first command is present and a first time has passed after the determination of the first command based on the gesture operation. [3] The driving assistance device according to claim 1 or 2, wherein the image output unit outputs, to the display unit, an image in which a display mode of the vehicle object is different from a display mode during a period in which the operation signal input unit can accept the gesture operation of the user, during a period in which the operation signal input unit can accept the gesture operation of the user. [4] The driving assistance device according to claim 1 or 2, wherein the image output unit changes sizes of the vehicle object, the non-vehicle object constituting the predetermined destination, and a surrounding state of the vehicle so that an image including the predetermined destination other than the vehicle is displayed on the display unit. [5] A driving assistance method performed by a driving assistance device, comprising: a step of outputting an image including a vehicle object representing a vehicle and the surrounding state of the vehicle to a display unit; a step of receiving a gesture operation that moves the vehicle object in the image displayed on the display unit to a position of an object that is a predetermined destination other than the vehicle when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation by a user can be accepted; and a step of outputting a first command commanding a change in a relative positional relationship between the vehicle and the predetermined destination, or a second command commanding an output of a message from the vehicle to the predetermined destination, to an automatic driving control unit that controls the automatic driving when the user's gesture operation has been received, wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first or second command is present and a first time has passed after the determination of the first or second command based on the gesture operation. [6] A driving assistance method performed by a driving assistance device, comprising: a step of outputting an image including a vehicle object representing a vehicle and an environmental state of the vehicle to a display unit; a step of receiving a gesture operation that exchanges the vehicle object in the image displayed on the display unit and an object representing a predetermined destination other than the vehicle in the image when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation by a user can be accepted; and a step of outputting a first command commanding a change in a relative positional relationship between the vehicle and the predetermined destination to an automatic driving control unit that controls the automatic driving when the user's gesture operation has been received, wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first command is present and a first time has passed after the determination of the first command based on the gesture operation. [7] Driving assistance system, comprising: a display device that displays an image; and a driving support device that outputs an image to the display device, wherein the driving assistance device an image output unit that outputs an image including a vehicle object representing a vehicle and the surrounding state of the vehicle to the display device; an operation signal input unit that, when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation can be accepted by a user, receives a gesture operation that moves the vehicle object in the image displayed on the display device to a position of an object that represents a predetermined destination other than the vehicle; and a command output unit that, when the user's gesture operation is received, outputs a first command that commands a change in a relative positional relationship between the vehicle and the predetermined destination, or a second command that commands an output of a message from the vehicle to the predetermined destination, to an automatic driving control unit that controls the automatic driving, wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first or second command is present and a first time has passed after the determination of the first or second command based on the gesture operation. [8] Driving assistance system comprising: a display device that displays an image; and a driving support device that outputs an image to the display device, wherein the driving assistance device an image output unit that outputs an image including a vehicle object representing a vehicle and an environmental state of the vehicle to the display device; an operation signal input unit that, when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation can be accepted by a user, receives a gesture operation that exchanges the vehicle object in the image displayed on the display device and a vehicle object that represents a predetermined destination different from the vehicle in the image; and a command output unit that, when the user's gesture operation is received, outputs a first command commanding a change in a relative positional relationship between the vehicle and the predetermined destination to an automatic driving control unit that controls the automatic driving, has wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first command is present and a first time has passed after the determination of the first command based on the gesture operation. [9] Automatically driving vehicle, comprising: an image output unit that outputs an image including a vehicle object representing a vehicle and the surrounding state of the vehicle to a display unit; an operation signal input unit that, when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation by a user can be accepted, receives a gesture operation that moves the vehicle object in the image displayed on the display unit to a position of an object that is a predetermined destination other than the vehicle; a command output unit that, when the user's gesture operation is received, outputs a first command commanding a change in a relative positional relationship between the vehicle and the predetermined destination, or a second command commanding output of a message from the vehicle to the predetermined destination; and an automatic driving control unit that executes the issued command, wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first or second command is present and a first time has passed after the determination of the first or second command based on the gesture operation. [10] Automatically driving vehicle, comprising: an image output unit that outputs an image including a vehicle object representing a vehicle and an environmental state of the vehicle to a display unit; an operation signal input unit that, when an acceptance mode of the driving assistance device is set as an acceptance enable mode in which an operation by a user can be accepted, receives a gesture operation that exchanges the vehicle object in the image displayed on the display unit and an object representing a predetermined destination other than the vehicle in the image; a command output unit that, when the user's gesture operation is received, outputs a first command commanding a change in a relative positional relationship between the vehicle and the predetermined destination; and an automatic drive control unit that executes the issued command wherein the acceptance mode is changed from the acceptance enable mode to an acceptance disable mode in which no operation can be accepted by the user when the first command is present and a first time has passed after the determination of the first command based on the gesture operation. [11] The driving assistance device according to claim 1 or 2, wherein the predetermined target is another vehicle or a pedestrian. [12] The driving support method according to claim 5 or 6, wherein the predetermined target is another vehicle or a pedestrian. [13] A driving support system according to claim 7 or 8, wherein the predetermined target is another vehicle or a pedestrian. [14] An automatic driving vehicle according to claim 9 or 10, wherein the predetermined target is another vehicle or a pedestrian. [15] The driving assistance device according to claim 1, wherein the receiving mode is changed from the receiving inhibition mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode is changed to the receiving inhibition mode, (ii) a notification indicating the completion of the control corresponding to the first or second command is received from the automatic driving control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first or second command is completed. [16] A driving assistance device according to claim 2, wherein the receiving mode is changed from the receiving inhibition mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode is changed to the receiving inhibition mode, (ii) a notification indicating the completion of the control corresponding to the first command is received from the automatic driving control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first command is completed. [17] The driving support method according to claim 5, wherein the receiving mode is changed from the receiving inhibition mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode was changed to the receiving inhibition mode, (ii) a notification indicating the completion of the control corresponding to the first or second command is received from the automatic driving control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first or second command is completed. [18] The driving assistance method according to claim 6, wherein the receiving mode is changed from the receiving inhibition mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode was changed to the receiving inhibition mode, (ii) a notification indicating the completion of the control corresponding to the first command is received from the automatic driving control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first command is completed. [19] A driving support system according to claim 7, wherein the receiving mode is changed from the receiving inhibit mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode was changed to the receiving inhibit mode, (ii) a notification indicating the completion of the control corresponding to the first or second command is received from the automatic travel control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first or second command is completed. [20] A driving support system according to claim 8, wherein the receiving mode is changed from the receiving inhibition mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode was changed to the receiving inhibition mode, (ii) a notification indicating the completion of the control corresponding to the first command is received from the automatic travel control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first command is completed. [21] An automatic traveling vehicle according to claim 9, wherein the receiving mode is changed from the receiving inhibit mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode was changed to the receiving inhibit mode, (ii) a notification indicating the completion of the control corresponding to the first or second command is received from the automatic traveling control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first or second command is completed. [22] An automatic traveling vehicle according to claim 10, wherein the receiving mode is changed from the receiving inhibit mode to the receiving enable mode when (i) a second time has elapsed after the receiving mode was changed to the receiving inhibit mode, (ii) a notification indicating the completion of the control corresponding to the first command is received from the automatic traveling control unit, or (iii) it is determined based on the behavior of the vehicle that the control corresponding to the first command is completed.

Citation Information

Patent Citations

  • Procedures for assisting a driver of a motor vehicle

    DE102010030463A1

  • Driver assistance system for motor car, has user interface the enables the driver to execute several functions of assistance system as function of specified objects in environment

    DE102011087459A1

  • Method for navigating an autonomous vehicle using a graphical projection display

    DE102011121746A1

  • Self-traveling vehicle

    JP1998105885A

  • JP000H10105885A