Driving assistance device and driving assistance method

The driving support device addresses the lack of clear operation timing and details in existing systems by providing drivers with precise maneuver information, improving obstacle avoidance capabilities.

JP7596187B2Active Publication Date: 2024-12-09NISSAN MOTOR CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2021042800
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-16
Publication Date
2024-12-09
Estimated Expiration
2041-03-16

AI Technical Summary

Technical Problem

Existing driving support systems fail to provide drivers with specific operation timing and details for lateral vehicle maneuvers to avoid obstacles, leaving them uncertain about when and how to execute these maneuvers.

Method used

A driving support device that acquires the driving situation ahead of the vehicle, determines the need for operation changes, and generates information on operation timing and content, which is then transmitted to the vehicle for display to the driver.

Benefits of technology

Enables drivers to grasp specific operation timing and details for vehicle maneuvers, enhancing their ability to navigate obstacles effectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007596187000001
    Figure 0007596187000001
  • Figure 0007596187000002
    Figure 0007596187000002
  • Figure 0007596187000003
    Figure 0007596187000003
Patent Text Reader

Abstract

To provide a driving support device and a driving support method which enable a driver to grasp a specific operation timing and specific operation contents of a vehicle.SOLUTION: The driving support device obtains a running situation around a target position in front of a vehicle; determines whether it is necessary to change driving-operation contents which are a type of driving operation of the vehicle to driving-operation contents which are different from current driving-operation contents or not, at the target position, on the basis of the running situation; and when determining that it is necessary to change the driving-operation contents, generates driving-operation change information showing driving-operation contents at the target position and timing information showing a timing at which the driving-operation contents are changed, and generates a control signal for making an output device provided in a vehicle interior of the vehicle output the driving-operation change information and the timing information.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a driving assistance device and a driving assistance method. [Background technology]

[0002] 2. Description of the Related Art There is known a vehicle information providing device that sets a route along which a vehicle should travel and provides route guidance, and that displays an image of a virtual lead vehicle within the real scenery that the driver is viewing.

[0003] The technology described in Patent Document 1 determines that, when there is a possibility that a vehicle will collide with an obstacle, an avoidance action is necessary for the obstacle, and displaces the display position of the image of the virtual lead vehicle laterally so as to avoid the obstacle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-107917 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology of Patent Document 1, although the driver can understand that it is necessary to move the vehicle laterally to avoid an obstacle, there is a problem in that the driver cannot understand the specific operation timing and specific operation content, such as what operation to perform and when to perform in order to move the vehicle laterally.

[0006] An object of the present invention is to provide a driving assistance device and a driving assistance method that enable a driver to grasp the specific operation timing and specific operation content of a vehicle. [Means for solving the problem]

[0007] The present invention solves the above problem by acquiring driving conditions around a target position in front of the vehicle, determining based on the driving conditions whether or not it is necessary to change the driving operation content, which is a type of driving operation of the vehicle, at the target position to driving operation content different from the current driving operation content, and if it is determined that the driving operation content needs to be changed, generating driving operation change information indicating the driving operation content at the target position and timing information indicating the timing for changing the driving operation content, and generating a control signal to output the driving operation change information and the timing information to an output device provided in the passenger compartment of the vehicle. Effect of the Invention

[0008] According to the present invention, the driver can grasp the specific operation timing and the specific operation content of the vehicle. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a driving assistance system according to this embodiment. [Figure 2A] FIG. 2A is a diagram showing an example of the position of a virtual vehicle in the environment surrounding a vehicle 200. As shown in FIG. [Figure 2B] FIG. 2B is a diagram showing an example of the position of a virtual vehicle in the environment surrounding the vehicle 200. [Diagram 3] FIG. 3 is a diagram showing an example of a display of driving operation change information and timing information generated by the driving support device of this embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of a procedure of the driving assistance method according to this embodiment. [Diagram 5] FIG. 5 is a flowchart showing an example of a procedure of the driving assistance method according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] First Embodiment An embodiment of the driving assistance device according to the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of a driving assistance system 10 including a driving assistance device 100 according to the present embodiment. As shown in FIG. 1, in the present embodiment, the driving assistance system 10 includes a driving assistance device 100, a vehicle 200, a second driving operation device 300, and a second information output device 310. The driving assistance device 100 is a server located away from the vehicle 200 and capable of communicating with the vehicle 200 and exchanging information. The driving assistance device 100 transmits driving assistance information to the vehicle 200 for supporting a first user who manually drives the vehicle 200. The driving assistance device 100 also transmits information on the driving situation at a position ahead of the vehicle 200 in the traveling direction to a second user who provides driving assistance to the first user. The remote space in which the driving assistance device 100 is installed is, for example, a control center that manages a plurality of vehicles. The second user is, for example, an operator of the control center. In the control center, the second information output device 310 outputs information on the driving conditions around the position in front of the vehicle 200 to the second user.

[0011] The second user provides the first user with driving assistance information related to the driving operation of the vehicle 200 while checking information on the driving situation at a position ahead of the vehicle 200. For example, one method of providing the driving assistance information to the first user is to display a virtual vehicle image representing a virtual vehicle driving ahead of the vehicle 200 on the windshield of the vehicle 200 so as to be superimposed on the actual scene outside the vehicle. The second user leads the vehicle 200 by operating the virtual vehicle, and provides driving guidance to the first user according to the behavior of the virtual vehicle. Since the second user operates the virtual vehicle in response to changes in the driving situation while watching the surrounding environment at a position ahead of the position of the vehicle 200 in the traveling direction, the second user can provide the first user with driving content required at a position ahead of the traveling direction of the vehicle 200.

[0012] On the other hand, the first user operates the vehicle 200 while watching the movement of the virtual vehicle displayed on the windshield of the vehicle 200. In addition, as a method of providing driving support information to the first user, when it is necessary to change the driving operation content, there is a method of displaying images of driving operation change information and timing information on the windshield of the vehicle 200. The driving operation change information is information indicating the specific driving operation content. In addition, the timing information is information indicating the timing to change the driving operation content. In this embodiment, when it is necessary to change the driving operation content at a position in front of the vehicle 200, an image indicating the specific driving operation content and the timing to change the driving operation content is displayed to the first user. When the first user is a beginner driver, it may be difficult to know what driving operation to perform at what timing just by looking at the behavior of the virtual vehicle, so in this embodiment, the driving of the first user is supported by presenting the content of the driving operation and the timing to perform the driving operation to the first user. In addition, in this embodiment, the driving support information is not limited to an image and may be a sound.

[0013] The driving support device 100 includes a controller 110 and a communication device 120. The controller 110 includes a computer having hardware and software, and the computer includes a ROM (Read Only Memory) that stores a program, a CPU (Central Processing Unit) that executes the program stored in the ROM, and a RAM (Random Access Memory) that functions as an accessible storage device. As the operating circuit, a MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like can be used instead of or in addition to the CPU. The driving support device 100 is not limited to being installed in a remote space where the second user is located, and may be installed in a different space.

[0014] The controller 110 includes at least a target position setting unit 111, a driving situation acquisition unit 112, a virtual information generation unit 113, a driving change necessity determination unit 114, a guidance information generation unit 115, a timing information generation unit 116, and a control signal generation unit 117 as functional blocks, and executes each function by cooperation between software for implementing each function or executing each process and hardware. Specifically, the controller 110 first sets a target position ahead in the traveling direction of the vehicle 200, and acquires the driving situation of the surrounding environment of the target position. Next, the controller 110 uses the information of the driving situation to generate virtual surrounding information representing the surrounding environment of the target position, and transmits a control signal for causing the second information output device 310 to output the virtual surrounding information. Then, the controller 110 determines whether or not it is necessary to change the driving operation content at the target position based on the driving situation at the target position. When it is determined that it is necessary to change the driving operation content, the controller 110 generates a control signal for causing the first information output device 240 to output the driving operation change information generated based on the driving operation information. Moreover, the controller 110 generates a control signal for outputting timing information to the first information output device 240. In this embodiment, the functions of the controller 110 are divided into seven blocks, and the functions of each functional block are described, but the functions of the controller 110 do not necessarily have to be divided into seven blocks, and may be divided into six or less functional blocks, or eight or more functional blocks.

[0015] The controller 110 displays a virtual vehicle image representing a virtual vehicle so as to be superimposed on a target position in the actual scene of the driving environment around the vehicle 200. In addition, when the second user inputs driving operation information, the controller 110 updates the target position based on the driving operation information, and displays the virtual vehicle so as to be superimposed on the updated target position, thereby updating the display position of the virtual vehicle. FIG. 2A is a diagram showing the position of a virtual vehicle on the surrounding environment of the vehicle 200 in this embodiment. In FIG. 2A, the virtual vehicle Va is located ahead of the vehicle 200 in the traveling direction. Vehicles V1, V2, and V3 are vehicles that are actually running around the vehicle 200. On the other hand, the virtual vehicle Va is displayed to the first user as if the virtual vehicle Va is running on the actual scene around the vehicle 200 seen from the first user of the vehicle 200. At this time, the second user performs a driving operation of the virtual vehicle Va while viewing a virtual image showing the actual scene around the virtual vehicle Va seen from the position of the virtual vehicle Va. FIG. 2A shows a scene in which the second user drives the virtual vehicle Va to change lanes to the adjacent lane on the left side when the virtual vehicle Va is displayed as traveling in the center lane. FIG. 2B shows a scene after the second user executes a driving operation related to lane change of the virtual vehicle Va. When the second user executes a driving operation of changing lanes of the virtual vehicle Va to the adjacent lane, the target position is updated to the adjacent lane according to the driving operation, and the virtual vehicle Va is displayed at the target position on the adjacent lane. As a result, in the field of view of the first user, the virtual vehicle Va appears to move from the center lane to the adjacent lane and travel on the adjacent lane. In addition, the second user performs a driving operation of the virtual vehicle Va while viewing a virtual surrounding image representing the actual scenery around the virtual vehicle seen from the position of the virtual vehicle Va shown in FIG. 2B. A more specific control method will be described below.

[0016] The target position setting unit 111 sets a target position ahead of the vehicle 200 in the traveling direction. The target position is the position of a virtual viewpoint set in the surrounding environment of the vehicle 200. The target position is also the position of a virtual vehicle in the surrounding environment of the vehicle 200. For example, the target position is a position corresponding to the position of the driver's seat of the virtual vehicle. The second user can see the scenery of the surrounding environment of the virtual vehicle from the target position. The target position is set as a coordinate position in the real space based on the position of the vehicle 200. In this embodiment, when the first user starts the virtual leading vehicle system, the target position setting unit 111 sets the target position to an initial setting position. The initial setting position is a predefined position, for example, a position ahead of the traveling direction of the vehicle 200 and a predetermined distance away from the vehicle 200.

[0017] Also, the target position setting unit 111 updates the target position based on the driving operation information input by the second user. The driving operation information is information for moving the virtual vehicle. The driving operation information includes information on the steering, accelerator, and brake operations of the vehicle. The driving operation information is output to the controller 110 via the second driving operation device 300 at a constant cycle. For example, the driving operation information is information on the steering operation output by the second user performing a steering operation. Also, for example, when the second user continues to maintain the vehicle speed of the virtual vehicle at a constant speed, the driving operation information is information on the accelerator operation for maintaining the vehicle speed at a constant speed. The target position setting unit 111 updates the target position corresponding to the position of the virtual vehicle based on the driving operation information for the virtual vehicle. Specifically, the target position setting unit 111 calculates the position of the target position after a predetermined time (for example, after one second) based on the position of the target position and the driving operation information, and updates the target position to the calculated target position after the predetermined time. For example, the target position setting unit 111 first calculates a moving distance and a moving direction after a predetermined time based on the vehicle speed information and steering angle information of the vehicle 200 acquired as the driving operation information. Next, the target position setting unit 111 calculates a position moved in the moving direction by the moving distance based on the current target position as the target position after the predetermined time. Then, the target position setting unit 111 updates the target position to the calculated target position after the predetermined time.

[0018] The driving condition acquisition unit 112 acquires driving conditions of the surrounding environment of the target position. Specifically, the driving condition acquisition unit 112 acquires a surrounding image of the target position captured by the captured image acquisition unit 222 of the vehicle 200. The driving condition acquisition unit 112 may acquire a surrounding image of the target position captured by a sensor group of another vehicle present in the vicinity of the target position. The driving condition acquisition unit 112 may acquire a surrounding image of the target position captured by an infrastructure sensor. The infrastructure sensor is a device that is set along the road on which the vehicle 200 is traveling and detects environmental information around the infrastructure sensor.

[0019] Furthermore, the traveling condition acquisition unit 112 acquires audio information in the surrounding environment of the target position. Specifically, the traveling condition acquisition unit 112 acquires audio information in the surroundings of the target position acquired by the external audio acquisition unit 223 of the vehicle 200. The traveling condition acquisition unit 112 may acquire audio information detected by a sensor group of another vehicle from another vehicle existing in the surroundings of the target position. The traveling condition acquisition unit 112 may acquire audio information in the surroundings of the target position detected by an infrastructure sensor. Furthermore, the traveling condition acquisition unit 112 is not limited to acquiring both a surrounding image and audio information, and may acquire only a surrounding image.

[0020] The virtual information generating unit 113 generates a virtual surrounding image that represents the surrounding environment of the target position. That is, the virtual surrounding image is an image that represents the surrounding environment at the position of the virtual vehicle. By displaying the virtual surrounding image, the second user can visually recognize the surrounding environment of the virtual vehicle as if he / she were sitting in the driver's seat of the virtual vehicle. The virtual information generating unit 113 first acquires three-dimensional space information of the surrounding environment based on the vehicle 200, calculates the position of the target position in the three-dimensional space of the surrounding environment, and generates a virtual surrounding image at the target position using the surrounding image of the target position. The three-dimensional space information is acquired as three-dimensional space information of the surrounding environment based on the position information of the vehicle 200 and the three-dimensional map information. In addition, the three-dimensional space information may be generated in real time by estimating the position of the vehicle 200 and generating a three-dimensional environmental map related to the surrounding environment of the vehicle 200 using SLAM based on the sensor information acquired by the sensor group 220 of the vehicle 200. Furthermore, the virtual information generating unit 113 may generate a virtual surrounding image at the target position by a light field rendering technique. Specifically, the virtual information generating unit 113 acquires light ray information of the surrounding environment from surrounding images of the surrounding environment of the target position captured by cameras with multiple viewpoints, and generates a virtual surrounding image by synthesizing an image at the target position based on the light ray information reaching the target position.

[0021] The virtual information generating unit 113 generates a virtual ambient sound. The virtual ambient sound is a virtual sound heard from the surrounding environment at a target position. The virtual information generating unit 113 sets the position of a virtual sound source to correspond to the position of a sound source in the surrounding environment of the target position based on the sound information of the surrounding environment of the target position acquired by the driving condition acquiring unit 112, and performs audio signal processing of the virtual sound to localize a sound image of the virtual sound at the position of the virtual sound source. In this embodiment, the virtual information generating unit 113 may generate only a virtual ambient image without generating a virtual ambient sound.

[0022] Furthermore, the virtual information generating unit 113 generates a virtual vehicle image. The virtual vehicle image is configured, for example, by a 3D object of a vehicle. When the first user activates the virtual lead vehicle system, the virtual information generating unit 113 generates a virtual vehicle image. Furthermore, in this embodiment, the virtual information generating unit 113 is not limited to generating an image of a virtual vehicle, and may generate a virtual image showing other objects or figures. For example, the virtual information generating unit 113 may generate a virtual image of an arrow showing the traveling direction of the vehicle 200.

[0023] The driving change necessity determination unit 114 determines whether or not the driving operation content of the vehicle 200 needs to be changed to a driving operation content different from the current driving operation content at the target position. The driving operation content is a type that classifies the driving operation of the vehicle into a plurality of categories. The driving operation content includes a "steady driving operation", a "turning operation", an "acceleration operation", and a "deceleration operation". The "steady driving operation" is an operation for driving the vehicle while maintaining a certain range of speed and traveling direction. The "turning operation" is an operation for changing the traveling direction of the vehicle. For example, a steering operation of a predetermined operation amount or more is classified as a "turning operation". The "acceleration operation" is an operation for increasing the vehicle speed of the vehicle. For example, an accelerator operation of a predetermined depression amount or more is classified as an "acceleration operation". The "deceleration operation" is an operation for decreasing the vehicle speed of the vehicle. The "deceleration operation" includes an operation for stopping the vehicle. For example, a brake operation of a predetermined depression amount or more is classified as a "deceleration operation".

[0024] In this embodiment, the driving change necessity determination unit 114 determines whether or not it is necessary to change the driving operation content at the target position to a driving operation content different from the current driving operation content, depending on whether or not an object exists ahead of the target position. That is, the driving change necessity determination unit 114 determines whether or not it is necessary to change the driving operation content when the vehicle 200, which is traveling steadily, reaches the target position. First, the driving change necessity determination unit 114 determines whether or not an object exists ahead of the target position in the traveling direction of the virtual vehicle, based on the traveling situation at the target position acquired by the traveling situation acquisition unit 112. The object is something that affects the driving operation content of the vehicle 200, and includes, for example, a parked vehicle, a pedestrian, a construction site, a traffic light, and a preceding vehicle. Then, when an object exists ahead of the target position, the driving change necessity determination unit 114 determines that it is necessary to change the driving operation content of the vehicle 200 at the target position to a driving operation content different from the current driving operation content. For example, when the vehicle 200 is traveling steadily, the driving change necessity determination unit 114 determines that the driving operation needs to be changed from a "steady driving operation" to a "turning operation" when an object such as a parked vehicle, a pedestrian, or a construction site is present in front of the target position. Also, when the vehicle 200 is traveling steadily, the driving change necessity determination unit 114 determines that the driving operation needs to be changed from a "steady driving operation" to a "deceleration operation" when a red light is present in front of the target position or when a preceding vehicle that is about to stop is present. Also, in this embodiment, the driving operation is divided into four types, but the invention is not limited to this, and may be divided into five or more types or three or less types.

[0025] Further, the driving change necessity determination unit 114 may determine whether or not it is necessary to change the driving operation content of the vehicle 200 to a driving operation content different from the current driving operation content at the target position based on the driving operation information of the first user and the driving operation information of the second user. Specifically, the driving change necessity determination unit 114 identifies the current driving operation content of the vehicle 200 based on the driving operation information of the first user, and identifies the driving operation content at the target position based on the driving operation information of the second user. For example, the driving operation content is identified as a "steady driving operation", "turning operation", "acceleration operation", or "deceleration operation" depending on whether the operation amount of the steering wheel and the pedal acquired from the driving operation information satisfies a predetermined condition. Then, the driving change necessity determination unit 114 compares the current driving operation content with the driving operation content at the target position, and when the driving operation content is different, determines that it is necessary to change the driving operation content of the vehicle 200 to a driving operation content different from the current driving operation content at the target position. In other words, when a second user driving a virtual vehicle based on the driving conditions at the target location performs a driving operation content different from the current driving operation content of vehicle 200 performed by the first user, the driving change necessity determination unit 114 determines that the driving operation content of vehicle 200 at the target location needs to be changed to a driving operation content different from the current driving operation content.

[0026] The guidance information generating unit 115 generates driving operation change information indicating the driving operation content at the target position. Specifically, the guidance information generating unit 115 first identifies the driving operation content at the target position based on the driving operation information of the second user at the target position. Then, the guidance information generating unit 115 generates driving operation change information indicating the identified driving operation content. For example, the guidance information generating unit 115 first acquires information on the steering operation amount and the pedal depression amount as the driving operation information of the second user at the target position, and compares the steering operation amount and the pedal depression amount with a predetermined condition set in advance for each driving operation content, thereby identifying the driving operation content that satisfies the predetermined condition as the driving operation content at the target position. The driving operation content includes the above-mentioned "steady driving operation", "turning operation", "acceleration operation", and "deceleration operation". Furthermore, the "turning operation" is identified as the steering direction, either left or right.

[0027] As a method of identifying the driving operation content, the guidance information generating unit 115 may identify the driving operation content suitable for the driving situation based on the driving situation. Specifically, first, the guidance information generating unit 115 determines the driving scene at the target position based on the driving situation acquired by the driving situation acquiring unit 112. The driving scene is set in advance as the driving scene corresponding to the driving situation. Then, the guidance information generating unit 115 identifies the driving operation set in advance for each driving scene as the driving operation content at the target position. The guidance information generating unit 115 may define a plurality of driving operation contents as states of a Markov model, and estimate the state using the Markov model with the driving operation information of the second user as a feature quantity, thereby identifying the driving operation content at the target position. The guidance information generating unit 115 may identify the driving operation content at the target position using a learned model that inputs the driving operation information of the second user as input data and outputs the driving operation content as output data. In addition, the guidance information generation unit 115 may construct a trained model by training a neural network using teacher data in which the input data is the driving operation information of the second user and the output data is the driving operation content, and use the constructed trained model to identify the driving operation content at the target location.

[0028] The guidance information generating unit 115 generates an image representing the driving operation content as driving operation change information indicating the specified driving operation content. Specifically, the guidance information generating unit 115 generates an image preset for each driving operation content according to the specified driving operation content. For example, if the specified driving operation content is a "turning operation", the guidance information generating unit 115 generates an image of a steering wheel indicating a steering operation and an image of an arrow indicating a turning direction. At this time, if the steering operation is to the right, an image of a steering wheel and an arrow pointing to the right is generated. The guidance information generating unit 115 may also generate an image indicating information on a target steering angle and a target vehicle speed that are targets of the driving operation. The target steering angle and the target vehicle speed are calculated from the steering operation amount and the pedal depression amount acquired as the driving operation information of the second user. For example, the guidance information generating unit 115 generates an image that displays the vehicle speed calculated from the pedal depression amount of the second user as the target vehicle speed. FIG. 3 is a diagram for explaining an example of driving operation change information and timing information generated in this embodiment. FIG. 3 is a diagram showing a scene in which a pedestrian Wa is present in front of the virtual vehicle Va, and the second user is moving the virtual vehicle to the right to avoid the pedestrian Wa. In this scene, first, the driving change necessity determination unit 114 determines that the driving operation content needs to be changed at the target position because the pedestrian Wa, which is an object, is present in front of the position (target position) of the virtual vehicle. Then, based on the driving operation information of the second user, the guidance information generation unit 115 specifies that the driving operation content at the target position is a "rightward turning operation", and generates an image showing a steering wheel and turning to the right as driving operation change information, as shown in FIG. 3.

[0029] The timing information generating unit 116 generates timing information indicating the timing to change the driving operation content. For example, the timing information generating unit 116 generates an image representing the actual scenery of the surrounding driving environment at the target position as the timing information. Since the image represents the surrounding scenery that the first user can see when the vehicle 200 reaches the target position, the image is displayed to the first user in advance, so that the first user can know in advance how the surrounding driving environment will look when the vehicle 200 reaches the target position. Then, when the vehicle 200 actually reaches the target position and the actual scenery that is the same as the actual scenery indicated by the timing information is actually seen, the first user can know that the timing is the timing to execute the change of the driving operation content. In this embodiment, the timing information generating unit 116 generates an image representing the actual scenery of the surrounding driving environment at the target position using a captured image of the driving environment at the target position in a manner similar to the generation of the virtual surrounding image by the virtual information generating unit 113. In addition, when an object exists in front of the target position, the timing information includes an image indicating the position and size of the object on the actual scenery of the driving environment around the target position. As a result, when an object in the real world appears in the field of view at the same position and size as the object in the displayed image, the first user can recognize that this is the timing to change the driving operation content.

[0030] FIG. 3 shows an example of an image of an object displayed as timing information. FIG. 3 shows a scene in which a pedestrian Wa is present ahead of the virtual vehicle Va in the traveling direction, and the virtual vehicle Va is moved to the right by the second user. In this scene, the timing information shown in FIG. 3 is displayed to the first user. The pedestrian Wa′ is the pedestrian Wa that appears in the image displayed as the timing information. The pedestrian Wa′ is displayed in the same position and size as when the pedestrian Wa is viewed from the position of the virtual vehicle Va. This allows the first user to understand that when the pedestrian Wa appears in the field of view at the same position and size as the displayed pedestrian Wa′, that timing is the timing to change the driving operation content.

[0031] Furthermore, when the first user performs a required driving operation at the target position, the timing information generating unit 116 generates an image representing the actual scenery of the driving environment around the target position based on the driving conditions at the target position acquired after the execution. Since the target position is updated based on the driving operation information of the second user, after the first user performs the required driving operation, the image representing the newly updated driving environment around the target position is displayed, so that the first user can grasp the driving environment around the current target position. Specifically, the timing information generating unit 116 generates an image representing the actual scenery of the driving environment around the target position using a captured image of the driving conditions at the target position acquired by the driving condition acquiring unit 112 after the execution.

[0032] Moreover, the control signal generating unit 117 generates a control signal for causing the first information output device 240 to display the virtual vehicle image so as to be superimposed on the target position set by the target position setting unit 111. Moreover, when the driving operation change information is generated by the guidance information generating unit 115, the control signal generating unit 117 generates a control signal for causing the first information output device 240 to output the driving operation change information. Moreover, when the timing information is generated by the timing information generating unit 116, the control signal generating unit 117 generates a control signal for causing the first information output device 240 to output the timing information. When the control signal is generated, the control signal generating unit 117 outputs the control signal to the communication device 120. Moreover, the control signal generating unit 117 may generate a control signal for causing the first information output device 240 to output voice information related to driving assistance based on voice information of the second user. Moreover, the control signal generating unit 117 generates a control signal for causing the second information output device 310 to output virtual surrounding information including information on the virtual surrounding image and virtual surrounding voice generated by the virtual information generating unit 113.

[0033] The communication device 120 communicates with the in-vehicle communication device 230 of the vehicle 200 to exchange information. Specifically, the communication device 120 receives peripheral information of the vehicle 200 from the in-vehicle communication device 230 and transmits driving assistance information to the in-vehicle communication device 230. Specifically, the communication device 120 transmits a control signal to the in-vehicle communication device 230 to cause the first information output device 240 to output a virtual vehicle image. In addition, the communication device 120 transmits a control signal to the in-vehicle communication device 230 to cause the first information output device 240 to output driving operation change information and timing information.

[0034] Next, the vehicle 200 will be described. The vehicle 200 includes an in-vehicle controller 210, a sensor group 220, an in-vehicle communication device 230, a first information output device 240, a first driving operation device 250, and a map database 260. The in-vehicle controller 210 controls the functions of each device and sensor of the vehicle 200. Specifically, the in-vehicle controller 210 acquires sensor information from the sensor group 220 and outputs it to the in-vehicle communication device 230. The in-vehicle controller 210 also outputs driving assistance information transmitted from the driving assistance device 100 to the first information output device 240.

[0035] The sensor group 220 acquires information on the state of the surrounding environment of the vehicle 200. The sensor group 220 includes a position information acquisition unit 221, a captured image acquisition unit 222, an external sound acquisition unit 223, an object detection unit 224, and a map information acquisition unit 225. The position information acquisition unit 221 detects the current position of the vehicle 200. The position information acquisition unit 221 is, for example, a GPS, and acquires the position information of the vehicle by receiving radio waves transmitted from a plurality of satellite communications with a receiver. The position information acquisition unit 221 can also detect changes in the position information of the vehicle by periodically receiving radio waves transmitted from a plurality of communication satellites. The captured image acquisition unit 222 captures an image of the driving environment around the vehicle 200. The captured image acquisition unit 222 is a well-known imaging device that includes an imaging element such as a so-called CCD or CMOS, an optical lens, an optical filter, a peripheral electronic circuit such as a power supply, and captures an image using the imaging element. The captured image acquisition unit 222 is arranged to capture images of the surrounding environment of the vehicle 200. For example, the captured image acquisition unit 222 is a plurality of cameras arranged on the front, rear, left and right sides of the vehicle 200, and captures images of scenery outside the vehicle in various directions. The captured image acquisition unit 222 may acquire a surrounding image including depth information. Furthermore, the captured image acquisition unit 222 repeatedly captures the surrounding image at a predetermined cycle. The captured image acquisition unit 222 transmits the captured captured images to the in-vehicle controller 210.

[0036] The external sound acquisition unit 223 collects sounds in the surrounding environment of the vehicle 200. The external sound acquisition unit 223 is, for example, a microphone array in which a plurality of microphones are set on the outside of the vehicle. The external sound acquisition unit 223 acquires spatial information of the sound and estimates the direction of the sound source. The external sound acquisition unit 223 transmits surrounding sound information obtained by converting the collected surrounding sounds into audio signals to the in-vehicle controller 210. The object detection unit 224 detects objects around the vehicle 200. For example, the object detection unit 224 irradiates the periphery of the vehicle 200 with millimeter waves or ultrasonic waves to scan a predetermined surrounding range and detects objects present around the vehicle 200. The objects include other vehicles, bicycles, pedestrians, traffic restrictions, construction sites, and the like present around the vehicle 200. The object detection unit 224 detects the relative position (orientation) between the object and the vehicle 200 and the distance between the object and the vehicle 200. The map information acquisition unit 225 acquires map information including the surrounding environment of the vehicle 200. Specifically, the map information acquisition unit 225 acquires map information including the surrounding environment of the vehicle 200 from the map database 260 based on the position information of the vehicle 200 acquired by the position information acquisition unit 221. The map information acquisition unit 225 may acquire map information from an external map database. The map information acquired by the map information acquisition unit 225 is output to the in-vehicle controller 210.

[0037] The in-vehicle communication device 230 communicates with the communication device 120 to exchange information. Specifically, the in-vehicle communication device 230 receives driving assistance information from the communication device 120, and transmits sensor information and map information to the communication device 120.

[0038] The first information output device 240 outputs image information and audio information to the first user. The first information output device 240 includes an image display device 241 and an audio output device 242. The first information output device 240 is, for example, an AR head-mounted display. The first user can view an augmented reality (AR) in which an image of a virtual vehicle is superimposed on a real scene via the first information output device 240. The image display device 241 displays a virtual vehicle image so as to be superimposed on a target position on the real scene of the surrounding environment of the vehicle 200. The image display device 241 also displays an image indicating driving operation change information and timing information. The image display device 241 is, for example, a transmissive display. The image display device 241 is not limited to a display of an AR head-mounted display, and may be the windshield of the vehicle 200. The audio output device 242 may also output audio information related to driving assistance by the second user. The audio output device 242 is, for example, a headphone provided in an AR head mounted display or a speaker installed in the vehicle 200.

[0039] The first driving operation device 250 is used by the first user to input driving operation information of the vehicle 200. The first driving operation device 250 includes at least a steering wheel 251 and pedals 252. The pedals 252 include an accelerator pedal and a brake pedal. The first user operates the steering wheel 251 and the pedals 252, whereby the driving operation information is input to the first driving operation device 250. For example, the first user operates the steering wheel 251 and the pedals 252, whereby driving operation information including steering angle information and pedal depression amount information is input to the first driving operation device 250, and the traveling of the vehicle 200 is controlled based on the input driving operation information. In addition, the driving operation information input by the first user is transmitted to the driving assistance device 100 via the in-vehicle communication device 230.

[0040] The map database 260 stores map information including road information and the like in advance. The map database 260 includes, for example, two-dimensional map information and three-dimensional map information. The three-dimensional map information is map information including three-dimensional information such as land elevation and building height. The three-dimensional map information may be information including a three-dimensional shape model of the terrain or buildings on the map, or information including a model in which an image of the terrain or buildings is added as texture to the shape model.

[0041] The second driving operation device 300 is used by the second user to input driving operation information of the virtual vehicle. The second driving operation device 300 includes a steering wheel 301 and pedals 302. The second driving operation device 300 includes, for example, the steering wheel 301 and pedals 302, which are shaped and structured like a steering wheel or an accelerator pedal of a general vehicle. The second user operates the steering wheel 301 and pedals 302, so that the driving operation information is input to the second driving operation device 300. The driving operation information includes steering angle information and pedal depression amount information for moving the virtual vehicle. For example, the second user operates the steering wheel of the control device, so that the steering angle information is input to the second driving operation device 300. The second driving operation device 300 may also receive input of voice information as the driving operation information. For example, the second driving operation device 300 includes a microphone that collects the voice uttered by the second user. When the second user utters a voice, the voice information of the second user is input to the second driving operation device 300.

[0042] The second information output device 310 is a device that outputs information on the driving conditions around a target position to the second user. The second information output device 310 is, for example, a glasses-type or goggles-type VR head-mounted display, and is worn on the head of the second user. The second user can view a virtual reality (VR) in which the driving conditions around a virtual vehicle set at the target position are displayed. The second information output device 310 outputs a virtual surrounding image that represents the driving conditions around the target position and virtual surrounding audio information that represents the audio around the target position. This allows the second user to see a view as if he or she were riding in a virtual vehicle, and to hear audio as if he or she were riding in a virtual vehicle. The second information output device 310 includes an image display device 311 and an audio output device 312. The image display device 311 is, for example, a non-transparent display. The audio output device 312 is, for example, a headphone or a speaker. The image display device 311 displays a virtual surrounding image generated by the virtual information generation unit 113. This allows the second user to view the surrounding environment of the virtual vehicle at the target position through the display on which the virtual surroundings image is displayed. The audio output device 312 outputs virtual surroundings audio information. This allows the second user to hear the audio around the virtual vehicle at the target position through headphones.

[0043] Next, the procedure of the driving assistance method in the driving assistance device according to this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing an example of the procedure of the driving assistance method according to this embodiment. Fig. 5 is a flowchart showing an example of the procedure of the driving assistance method according to this embodiment. The flowchart in Fig. 5 shows a flowchart of control in step S10 in Fig. 4. In this embodiment, when a first user of the vehicle 200 activates the virtual lead vehicle system and starts using it, control starts from step S1.

[0044] In step S1, the controller 110 sets the target position to an initial setting position. The initial setting position is a position ahead in the traveling direction of the vehicle 200 and at a predetermined distance from the vehicle 200. In step S2, the controller 110 displays a virtual vehicle image at the target position. Specifically, the controller 110 generates a control signal for displaying the virtual vehicle image at the target position, and transmits the generated control signal to the vehicle 200. The first information output device 240 of the vehicle 200 displays the virtual vehicle image to the first user based on the transmitted control signal.

[0045] In step S3, the controller 110 acquires the driving conditions at the target position. Specifically, the controller 110 acquires a surrounding image and surrounding sound of the driving environment around the target position detected by the sensor group 220 of the vehicle 200, the sensor group of the other vehicle, and the infrastructure sensor. In step S4, the controller 110 outputs virtual surrounding information to the second user via the second information output device 310. Specifically, the controller 110 generates a virtual surrounding image and a virtual surrounding sound as the virtual surrounding information based on the driving conditions at the target position acquired in step S3. Then, the controller 110 generates a control signal for causing the second information output device 310 to output the virtual surrounding image and the virtual surrounding sound, and outputs the generated control signal to the second information output device 310. The second information output device 310 outputs the virtual surrounding image and the virtual surrounding sound based on the control signal.

[0046] In step S5, the controller 110 acquires driving operation information of the virtual vehicle output from the second driving operation device 300. The driving operation information of the virtual vehicle is information input by the second user to the second driving operation device 300. In step S6, the controller 110 updates the target position based on the driving operation information received in step S5. In steps S7 to S9, the controller 110 performs the same processes as steps S2 to S4 on the updated target position, and outputs virtual surrounding information at the updated target position.

[0047] In step S10, the controller 110 performs control to determine whether or not an operation change is necessary. After the control in step S9, the controller 110 proceeds to a flow of step S101 in Fig. 5, which will be described later. When the flow of step S10 ends, the process returns to the flow in Fig. 4 and proceeds to step S11.

[0048] In step S11, the controller 110 judges whether the virtual lead car system has been terminated. If it is judged that the virtual lead car system has been terminated, the flow is terminated. If it is not judged that the virtual lead car system has been terminated, the flow returns to step S4, and the flow is repeated thereafter. In addition, in this embodiment, while the virtual lead car system is running, the control flow is repeatedly executed at a constant cycle. Specifically, the controller 110 acquires peripheral information of the target position at a constant cycle, and updates the virtual peripheral information based on the acquired peripheral information. This makes it possible to update the peripheral environment of the virtual vehicle in response to the peripheral environment that changes in real time due to the operation of the virtual vehicle by the second user. Similarly, the position of the virtual vehicle in the virtual vehicle image displayed to the first user is updated in real time according to the operation of the second user.

[0049] Next, the procedure of the driving assistance method according to this embodiment will be described with reference to Fig. 5. In this embodiment, after the control of step S9 in Fig. 4, the control flow is executed from step S101 in Fig. 5.

[0050] In step S101, the controller 110 judges whether or not the driving operation content at the target position needs to be changed to a driving operation content different from the current driving operation content. Specifically, the controller 110 judges whether or not an object that affects the driving of the vehicle exists ahead of the target position based on the driving situation at the target position. Then, the controller 110 judges whether or not the driving operation content at the target position needs to be changed to a driving operation content different from the current driving operation content, depending on the judgment result regarding the presence or absence of the object. If it is judged that the driving operation content needs to be changed, the process proceeds to step S102. If it is judged that the driving operation content does not need to be changed, the flow in step S10 is terminated, and the process proceeds to step S11 in FIG. 4.

[0051] In step S102, the controller 110 generates driving operation change information indicating the driving operation content at the target position. For example, the controller 110 identifies the driving operation content at the target position based on the driving operation information input by the second user, and generates driving operation change information indicating the identified driving operation content.

[0052] In step S103, the controller 110 generates timing information indicating the timing to change the driving operation content. For example, the controller 110 generates an image showing the actual scene of the driving environment around the target position as the timing information.

[0053] In step S104, the controller 110 generates a control signal for causing the first information output device 240 to output the driving operation change information generated in step S102 and the timing information generated in step S103.

[0054] In step S105, the first information output device 240 of the vehicle 200 outputs the driving operation change information and timing information to the first user. Specifically, first, the controller 110 transmits the control signal generated in step S104 to the in-vehicle communication device 230. Next, the in-vehicle controller 210 outputs the control signal received by the in-vehicle communication device 230 to the first information output device 240. Then, the first information output device 240 outputs the driving operation change information and timing information to the first user based on the input control signal. After the control of step S105, the process returns to step S11 shown in FIG. 4. In this embodiment, whether or not a change in the driving operation content is necessary at the target position is determined at a certain period or each time the target position is updated, thereby determining whether or not a change in the driving operation is necessary according to the driving environment that changes in real time.

[0055] As described above, in this embodiment, the driving conditions around the target position in front of the vehicle are acquired, and based on the acquired driving conditions, it is determined whether or not it is necessary to change the driving operation content, which is a type of driving operation of the vehicle, to a driving operation content different from the current driving operation content at the target position, and if it is determined that the driving operation content needs to be changed, driving operation change information indicating the driving operation content at the target position is generated, timing information indicating the timing to change the driving operation content is generated, and a control signal is generated to output the driving operation change information and the timing information to an output device provided in the vehicle cabin. This allows the driver to grasp the specific operation timing and specific operation content of the vehicle.

[0056] In this embodiment, when the vehicle is located at a target position based on the driving situation, surrounding environment information showing the actual surrounding environment as seen from the viewpoint of the driver of the vehicle is generated as timing information. This allows the driver to know the timing to change the driving operation from the appearance of the surrounding environment when the vehicle reaches the target position.

[0057] In this embodiment, the driving situation is used to determine whether an object that will affect the driving operation is present ahead of the target position, and the driving situation is determined based on the result of the determination as to whether an object that will affect the driving operation is present ahead of the vehicle, and the driving situation is used to determine whether the driving operation needs to be changed at the target position. If it is determined that the driving operation needs to be changed at the target position, image information is generated as timing information based on the driving situation, which represents the position and size of the object in the actual scene that is visible from the driver's position when the vehicle is at the target position. This allows the driver to know the timing to change the driving operation from the position and size of the object in the actual scene that is visible when the vehicle reaches the target position.

[0058] In addition, in this embodiment, the driving operation change information indicating the driving operation suitable for the driving situation is generated based on the driving situation, and the driving operation change information is output to the output device, so that the driver can know in advance the driving operation suitable for the target position.

[0059] In addition, in this embodiment, the driving operation change information is generated based on the driving operation information input by the user who is located away from the vehicle, so that the driver can know the driving operation content that is generated from the user's driving operation and is more advanced than the automatically generated driving operation content.

[0060] In addition, in this embodiment, when a driving operation content at a target location is executed, a driving situation after the execution is acquired, surrounding environment information is generated based on the acquired driving situation after the execution, and a control signal for causing an output device to output the generated surrounding environment information is generated, so that the driver can know the driving situation at the target location after the driving operation content at the target location is executed.

[0061] The above-described embodiments are described for the purpose of facilitating understanding of the present invention, and are not described for the purpose of limiting the present invention. Therefore, the elements disclosed in the above embodiments are intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]

[0062] 10. Driving assistance systems 100...Driving assistance device 110…Controller 111...Target position setting unit 112…Driving status acquisition unit 113…Virtual Information Generation Division 114…Operation change necessity determination unit 115...Guidance information generation unit 116... Timing information generation unit 117...Control signal generating unit 200…Vehicle 210…In-vehicle controller 220...Sensor group 240...first information output device 300...Second driving operation device 310...Second information output device

Claims

1. A driving assistance device that assists driving of a vehicle, A driving condition acquisition unit that acquires a driving condition around a target position in front of the vehicle; a driving change necessity determination unit that determines whether or not a driving operation content, which is a type of driving operation of the vehicle, needs to be changed to a driving operation content different from a current driving operation content at the target position based on the driving condition acquired by the driving condition acquisition unit; a guidance information generating unit that generates driving operation change information indicating the driving operation content at the target position when the driving change necessity determining unit determines that the driving operation content needs to be changed; a timing information generating unit that generates timing information indicating a timing for changing the driving operation content; a control signal generating unit that generates a control signal for outputting the driving operation change information and the timing information to an output device provided in a cabin of the vehicle; The timing information generation unit is a driving assistance device that generates, as the timing information, surrounding environment information that indicates an actual view of the surrounding environment as seen from the viewpoint of the driver of the vehicle when the vehicle is located at the target position based on the driving conditions.

2. The operation change necessity determination unit is determining whether or not an object that will affect the driving operation is present in front of the target position based on the driving situation; determining whether or not it is necessary to change the driving operation at the target location based on a determination result regarding whether or not an object that will affect the driving operation is present in front of the vehicle; 2. The driving assistance device according to claim 1, wherein when the driving change necessity determination unit determines that it is necessary to change the driving operation content at the target position, the timing information generation unit generates, as the timing information, image information representing the position and size of the object in an actual scene visible from the driver's position when the vehicle is located at the target position, based on the driving conditions.

3. The guidance information generation unit generates the driving operation change information indicating the driving operation content suitable for the driving situation based on the driving situation, The driving support device according to claim 1 , wherein the control signal generating unit causes the output device to output the driving operation change information.

4. The driving support device according to claim 1 , wherein the guidance information generating unit generates the driving operation change information based on driving operation information input by a user who is located away from the vehicle.

5. The driving situation acquisition unit acquires the driving situation after the driving operation content at the target position is executed when the driving operation content is executed, The timing information generation unit generates the surrounding environment information based on the driving condition after the execution acquired by the driving condition acquisition unit, The driving support device according to claim 1 , wherein the control signal generating unit generates a control signal for causing the output device to output the generated surrounding environment information.

6. A driving assistance method for assisting driving of a vehicle, comprising: Acquire a driving situation around a target position in front of the vehicle; Based on the acquired driving conditions, determine whether or not it is necessary to change a driving operation content, which is a type of driving operation of the vehicle, to a driving operation content different from a current driving operation content at the target position; When it is determined that the driving operation needs to be changed, generating driving operation change information indicating the driving operation at the target position; generating surrounding environment information indicating an actual view of the surrounding environment as seen from a viewpoint of a driver of the vehicle when the vehicle is located at the target position based on the driving situation, as timing information indicating a timing to change the driving operation content; A driving assistance method for generating a control signal for outputting the driving operation change information and the timing information to an output device provided in a passenger compartment of the vehicle.

Citation Information

Patent Citations

  • Vehicular information providing device, vehicular information providing method, and vehicular information providing program

    JP2005107917A

  • Traffic information guide system and traffic information distribution device

    JP2019139349A

  • Vehicular display control device, vehicular display control method and vehicular display control program

    JP2020152161A