Driver assistance device, driver assistance system, and driver assistance method

Abstract

To provide a driver assistance device, driver assistance system, and driver assistance method that contribute to further improving the driving skills of drivers. [Solution] The system includes an environment information acquisition unit 101 that acquires information about a moving object and information about the surrounding environment of the moving object; an operation information identification unit 102 that identifies operation information, which is information about actual operations performed by the driver, based on the surrounding environment information; an exemplary operation calculation unit 104 that calculates exemplary operations that the driver should perform based on the surrounding environment information; a transition operation calculation unit 105 that uses the operation information and exemplary operations to calculate exemplary operation information that transitions from actual operations to exemplary operations; a virtual space construction unit 106 that virtually constructs the space in which the operation was performed based on the surrounding environment information; and an output unit 107 that outputs actual operation and exemplary operation information together with the space.

Description

Technical Field

[0001] The present invention relates to a driving support device, a driving support system, and a driving support method for improving the driving skills of a driver who drives an automobile.

Background Art

[0002] As background art in this technical field, there is Patent Document 1. Patent Document 1 describes that "it is possible to provide an effective driving training for an employee who is a trainee. The driving training system includes a server device and a first terminal device that communicates with the server device. The server device includes an acquisition unit that acquires driving data of a vehicle, an evaluation unit that evaluates the driving skills of an employee who has driven the vehicle based on the driving data acquired by the acquisition unit, and a selection unit that selects a driving training for the employee to attend based on the evaluation by the evaluation unit. The first terminal device superimposes or arranges and displays a driving video of the vehicle driven by the employee and a model video that is a model video corresponding to the evaluation items evaluated by the evaluation unit in the driving training selected by the selection unit, and synchronizes the reproduction start timing of the driving video and the reproduction start timing of the model video to reproduce the driving video and the model video."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional driving support device, the driving skills of a driver are evaluated from driving data acquired when the driver actually drives, and a model video that the driver should watch is selected based on this evaluation.

[0005] The aforementioned Patent Document 1 describes a method for providing effective driving instruction to drivers by placing a driving video, which was simultaneously filmed while the driver was driving, alongside a model video and playing them back in sync.

[0006] In many cases, the selected example video does not match the actual traffic conditions in the situation the driver is experiencing.

[0007] Therefore, Patent Document 1 requires improvement because it is difficult to adequately demonstrate, through example videos, what actions should be taken and when, in traffic situations experienced by the driver themselves.

[0008] Furthermore, the timing of steering wheel, accelerator, and brake operation in demonstration videos often does not match the timing of actual driver operation, so improvement is needed.

[0009] Furthermore, even if the start times of the demonstration video and the driving video are synchronized, it is suspected that timing discrepancies in operations may occur midway through. Consequently, it becomes difficult for drivers to efficiently learn the demonstration operations, and therefore improvements are needed.

[0010] The present invention provides a driver assistance device, a driver assistance system, and a driver assistance method that contribute to further improvement of drivers' driving skills. [Means for solving the problem]

[0011] The present invention includes multiple means for solving the above problems, but to give one example, it includes: an environment information acquisition unit that acquires mobile information relating to a mobile body and information about the surrounding environment of the mobile body; an operation information identification unit that identifies operation information which is information relating to actual operations performed by a driver based on the surrounding environment information; an exemplary operation calculation unit that calculates exemplary operations that the driver should perform based on the surrounding environment information; a transition operation calculation unit that calculates exemplary operation information which transitions from the actual operation to the exemplary operation using the operation information and the exemplary operation; a virtual space construction unit that virtually constructs a space in which the operation was performed based on the surrounding environment information; and an output unit that outputs the operation information and the exemplary operation information together with the space. [Effects of the Invention]

[0012] The present invention can contribute to further improvement of drivers' driving skills. Other issues, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram showing the configuration of the driver assistance device in Example 1. [Figure 2] This diagram illustrates the hardware configuration of the driver assistance device in Example 1. [Figure 3] This diagram illustrates the processing flow in the driver assistance device of Example 1. [Figure 4] This diagram illustrates the process for determining a vehicle's lane change in the driver assistance system of Example 1. [Figure 5] This figure shows the exemplary operation information held by the exemplary operation database in the driver assistance system of Example 1. [Figure 6] This figure illustrates the change in the direction of vehicle movement in the driver's own operation and the exemplary operation of the driver assistance device in Example 1. [Figure 7] This diagram illustrates the flow of the trajectory from the vehicle's trajectory through the transition trajectory to the model trajectory and the corresponding control input in the driving assistance device of Example 1. [Figure 8]This is a diagram for explaining the flow of the trajectory from the trajectory of the host vehicle in the driving support device of the first embodiment, through the transition trajectory, to the exemplary trajectory, as well as the operation amount and the display of the original trajectory of the host vehicle. [Figure 9] This is a diagram for explaining the result of showing, from the driver's perspective, the flow of the trajectory from the trajectory of the host vehicle in the driving support device of the first embodiment, through the transition trajectory, to the exemplary trajectory, as well as the display of the trajectory of the host vehicle. [Figure 10] This is a configuration diagram of the driving support device of the second embodiment. [Figure 11] This is a diagram for explaining the flow of the trajectory from the trajectory of the host vehicle in the driving support device of the second embodiment, through the transition trajectory, to the exemplary trajectory, as well as the operation amount. [Figure 12] This is a diagram for explaining an example of displaying the driving pattern by the transition operation in the driving support device of the third embodiment from another perspective. [Figure 13] This is a diagram for explaining an example of calculating the trajectory and operation amount for coping with a new traffic situation by adding a virtual object in the driving support device of the fourth embodiment. [Figure 14] This is a diagram showing the information of the exemplary operation held in the exemplary operation database in the example of adding a virtual object in the driving support device of the fourth embodiment. [Figure 15] This is a diagram for explaining an example of displaying the operation content and trajectory by superimposing them on the perspective in the actual environment in the driving support device of the fifth embodiment.

Mode for Carrying Out the Invention

[0014] Hereinafter, based on the surrounding environment and driving content when the driver drives, the embodiments of the driving support device, driving support system, and driving support method of the present invention that contribute to the improvement of driving skills by clarifying improvement points by calculating and outputting the operation amount for transitioning from the actual driving operation of the driver to an exemplary operation will be described with reference to the drawings.

[0015] Note that the moving objects targeted by the driving support device, driving support system, and driving support method of the present invention include, for example, automobiles, trucks, motorcycles, etc. In the following embodiments, mainly vehicles such as automobiles will be described.

[0016] Here, the driving support systems of each of the embodiments described below are assumed to include a driving support device described in each embodiment and a moving body having a display (display device), but are not limited thereto.

[0017] Also, the driving support device of each embodiment will be described by taking the stand-alone type as an example, but it can also be configured to be provided in the moving body together with the driving support device and the display.

[0018] Also, in the drawings used in this specification, the same or corresponding components may be given the same or similar reference numerals, and repeated descriptions of these components may be omitted.

[0019] <Example 1> Example 1 of the driving support device, driving support system, and driving support method of the present invention will be described using FIGS. 1 to 9.

[0020] First, the overall configuration of the driving support device will be described using FIG. 1. FIG. 1 is a configuration diagram of the driving support device of this embodiment.

[0021] The driving support device 1 shown in FIG. 1 is composed of an environment information acquisition unit 101, an operation information identification unit 102, a model operation database 103, a model operation calculation unit 104, a transition operation calculation unit 105, a virtual space construction unit 106, an output unit 107, and the like.

[0022] The environment information acquisition unit 101 acquires vehicle information regarding the vehicle and the surrounding environment information of the vehicle. Specifically, the environment information acquisition unit 101 acquires information on the surrounding environment, information on the vehicle itself, and information on the operation content of the driver regarding a vehicle such as the vehicle driven by the driver, and outputs it to the operation information identification unit 102, the transition operation calculation unit 105, and the output unit 107. This environment information acquisition unit 101 preferably serves as the execution subject of an environment information acquisition step for acquiring vehicle information regarding the vehicle and the surrounding environment information of the vehicle.

[0023] The environmental information acquisition unit 101 acquires information about the surrounding environment, such as the shape of the road surface while driving, surrounding structures such as buildings, traffic signs such as speed limits, and the types and arrangements of traffic signals. This various information can be acquired using, for example, cameras, LiDAR, radar, etc.

[0024] Furthermore, information about the vehicle itself includes components such as position, speed, acceleration, and direction of movement. This various information can be obtained using GPS and sensors mounted on the vehicle itself.

[0025] Information about the driver's actions includes components such as the steering wheel angle set by the driver and the amount the accelerator and brake pedals are pressed. This various information can be acquired using sensors installed in the vehicle itself.

[0026] The operation information identification unit 102 identifies operation information, which is information about the actual operation performed by the driver, based on the surrounding environment information. Specifically, the operation information identification unit 102 analyzes the surrounding environment information output by the environment information acquisition unit 101 to identify what operation the driver is about to perform, and outputs the resulting operation information to the model operation calculation unit 104. Preferably, this operation information identification unit 102 is the entity that executes the operation information identification step, which identifies operation information, which is information about the actual operation performed by the driver, based on the surrounding environment information.

[0027] The exemplary operation database 103 stores exemplary operation amounts in driving. Specifically, the exemplary operation database 103 stores information about the surrounding environment, combinations of operation information, and the content of exemplary operations in those combinations, and outputs this information.

[0028] The exemplary operation calculation unit 104 calculates exemplary operations that the driver should perform based on the surrounding environment information. Specifically, the exemplary operation calculation unit 104 refers to the contents of the exemplary operation database 103 based on the surrounding environment information and operation information input from the operation information identification unit 102, calculates information on exemplary operations, and outputs it to the transition operation calculation unit 105. Preferably, this exemplary operation calculation unit 104 is the main entity that executes the exemplary operation calculation step, which calculates exemplary operations that the driver should perform based on the surrounding environment information.

[0029] Furthermore, this exemplary operation calculation unit 104 can calculate exemplary operations that the driver should perform based on surrounding environment information, and can also calculate exemplary operations based on both surrounding environment information and operation information.

[0030] The transition operation calculation unit 105 uses the operation information and the model operation to calculate model operation information that transitions from the actual operation to the model operation. Preferably, the transition operation calculation unit 105 calculates model operation information using model operation quantity information stored in the model operation database 103. Specifically, the transition operation calculation unit 105 calculates the content of the transition operation to transition from the driver's current operation to the model operation based on the surrounding environment information output by the environment information acquisition unit 101, the vehicle itself, the driver's operation content information, and the model operation information output by the model operation calculation unit 104, and outputs it to the output unit 107. Preferably, this transition operation calculation unit 105 is the main entity that executes the transition operation calculation step, which calculates model operation information that transitions from the actual operation to the model operation using the operation information and the model operation.

[0031] The virtual space construction unit 106 virtually constructs the space in which the operation was performed based on surrounding environment information. Specifically, it virtually constructs the digital space in which the operation was performed based on surrounding environment information. Preferably, this virtual space construction unit 106 is the entity that executes the virtual space construction step, which virtually constructs the space in which the operation was performed based on surrounding environment information.

[0032] The output unit 107 outputs operation information and exemplary operation information to an external display device in the space constructed by the virtual space construction unit 106. Preferably, the output unit 107 can also highlight the transition trajectory 601 (see Figure 7) that transitions from actual operation to exemplary operation. The display device to which the output unit 107 outputs can be, for example, a display on a vehicle or a user terminal held by the user, but it can be any type of display device.

[0033] Specifically, the output unit 107 displays the environment in the digital space using the surrounding environment information output by the environment information acquisition unit 101, and also outputs the transition operation content output by the transition operation calculation unit 105 overlaid on the environment in the space. Preferably, this output unit 107 is the main entity that executes the output step that outputs operation information and model operation information together with the space.

[0034] Figure 2 shows the hardware configuration for the processing described above. Figure 2 is a diagram illustrating the hardware configuration of the driver assistance device in Example 1.

[0035] The system's components, including the sensor 902, CPU 904, storage device 903, RAM 905, and display 906, are connected to the data bus 901 and transmit and receive data from each other through it.

[0036] Sensor 902 includes cameras, LiDAR, radar, GPS, etc., and outputs information about the surrounding environment, information about the vehicle itself, and information about the driver's actions.

[0037] The storage device 903 stores and outputs data output by the sensor 902 and information held by the model operation database 103.

[0038] CPU904 performs processing such as determining the operation content, calculating transition operations, and outputting screen display content based on the input information.

[0039] RAM905 inputs and outputs intermediate results of calculations performed by CPU904, assisting CPU904's processing.

[0040] The display 906 corresponds to the output unit 107 and consists of a liquid crystal display or a VR head-mounted display, and displays screen content such as the environment and transition operation details output by the CPU 904.

[0041] An example of the above-mentioned driver assistance system calculating transition operation content and outputting it overlaid on the environment in space will be explained below using Figures 3 to 9. In this example, the vehicle is assumed to be an automobile and is traveling on a public road.

[0042] First, the processing flow of the driver assistance device in this embodiment will be explained using Figure 3. Figure 3 is a diagram illustrating the processing flow in the driver assistance device of Embodiment 1.

[0043] In the driver assistance device of the present invention, when operation starts, the process shown in Figure 3 is performed first, specifically the process in step S201.

[0044] In step S201, the environmental information acquisition unit 101 acquires environmental information about the area around the vehicle as the driver operates the vehicle.

[0045] In this example, environmental information refers to the arrangement and shape of objects such as the shape of the road, surrounding traffic signs, signals, buildings and other features, and moving objects on or around the road. When acquiring environmental information, equipment such as cameras, radar, and LiDAR mounted on the vehicle is used. After this, the process moves to step S202.

[0046] In step S202, the environmental information acquisition unit 101 acquires information about the vehicle. These steps S201 and S202 correspond to the environmental information acquisition steps.

[0047] In this example, the vehicle information consists of components such as the change in the vehicle's position obtained by GPS, etc., and the change in numerical values ​​such as the vehicle's yaw, pitch, and roll detected by on-board sensors. This information is acquired and used while the driver is operating the vehicle. After this, the process moves to step S203.

[0048] In step S203, the environmental information acquisition unit 101 acquires the operation details performed by the driver. These operation details consist of numerical changes, such as the angle of the steering wheel and the amount the accelerator and brake pedals are pressed. After this, the process moves to step S204.

[0049] Steps S201 through S203 can be performed in any order and may be processed simultaneously, or steps S202 and S203 may be executed before steps S201 and S202.

[0050] In step S204, the operation information identification unit 102 determines what operations the driver performed after finishing driving. These steps S203 and S204 correspond to the operation information identification steps.

[0051] Specifically, based on the environmental information, vehicle information, and operation details acquired so far, it is determined whether the vehicle changed lanes while driving. The position of the vehicle at a certain time t, calculated from the vehicle information, is denoted as p(t), and the lane containing this position is represented as l(p(t)) using road information acquired from the environmental information. The lane containing the vehicle's position p(t+i) at another time (t+i) is denoted as l(p(t+i)). If l(p(t))≠l(p(t+i)), it is determined that a lane change occurred between time t and time (t+i). An example of this is shown in Figure 4. Figure 4 is a diagram illustrating the process for determining a vehicle's lane change in the driver assistance system of Embodiment 1.

[0052] In Figure 4, assume that the vehicle is moving from right to left on a three-lane road running in the left-right direction. At time t and time (t+i), the vehicle is at positions p(t) and p(t+i), respectively. Since p(t) is in lane L1, l(p(t))=L1. Similarly, since p(t+i) is in lane L2, l(p(t+i))=L2. In this example, l(p(t))≠l(p(t+i)), and it is determined that a lane change occurred between time t and time (t+i). If it is determined that a lane change occurred in this way, proceed to step S205. Otherwise, proceed to step S207.

[0053] Next, in step S205, the model operation calculation unit 104 determines whether the vehicle changed direction while driving, based on the environmental information, vehicle information, and operation details acquired so far. Based on the environmental information, if it is determined using the vehicle information that the vehicle turned counterclockwise and left the point where three or more roads connect, it is determined that a left turn was made. Similarly, if it is determined using the vehicle information that the vehicle turned clockwise and left the point, it is determined that a right turn was made. If it is determined that a right turn or a left turn, i.e., a change of direction was made, the process moves to step S206; otherwise, the process moves to step S208.

[0054] In step S206, the exemplary operation calculation unit 104 determines the type of exemplary operation to be obtained from the exemplary operation database 103 to be a change of direction, specifically a right turn or a left turn. After this, the process moves to step S209.

[0055] In step S207, the model operation calculation unit 104 determines that the type of model operation to be obtained from the model operation database 103 is speed update. After this, the process moves to step S209.

[0056] In step S208, the exemplary operation calculation unit 104 determines the type of exemplary operation to be obtained from the exemplary operation database 103 to be passing or changing lanes. After this, the process moves to step S209.

[0057] Next, in step S209, the exemplary operation calculation unit 104 extracts a model operation of the type that matches from the model operations held in the model operation database 103, based on the type of model operation defined in the processes up to this point. This step S209 corresponds to the model operation calculation step.

[0058] Figure 5 shows an example of a model operation held by the model operation database 103. Figure 5 is a diagram showing the model operation information held by the model operation database in the driving assistance device of Embodiment 1.

[0059] For example, based on the types defined in steps S206, S207, and S208, and the turning direction, speed changes, and lane changes obtained from the vehicle's own information, exemplary operations are extracted according to subcategories. Exemplary operations are a collection of changes in the amount of steering wheel angle, accelerator and brake pedal pressure over time, for operations that enable efficient and safe driving.

[0060] More specifically, the exemplary operation for a left turn during a change of direction is represented by the operation amount M0(T0) at the initial time point T0. M0(T0) can be represented by numerical values ​​for, for example, the steering wheel angle, the amount the accelerator pedal is pressed, and the amount the brake pedal is pressed, and can be represented as a 3D vector combining these values. Subsequently, the operation amount M0(T1) is stored at time point T1, and similarly, the operation amounts at each subsequent time point are stored. This array of operation amounts is represented as M0, and this is used as the exemplary operation. After this, the process moves to step S210.

[0061] Next, in step S210, the transition operation calculation unit 105 compares the exemplary operation extracted in step S209 with the actual operation performed by the driver. The comparison is made by comparing the position and direction of the vehicle when the exemplary operation is performed with the position and direction of the vehicle when the actual operation is performed by the driver.

[0062] Specifically, based on the operation details acquired from the environmental information acquisition unit 101, the temporal progression of the driver's actual operation is represented by numerical values ​​for the steering wheel angle, accelerator pedal depression amount, and brake pedal depression amount, similar to the exemplary operation, and the resulting three-dimensional vector is represented by m. The amount of operation at a certain time t is represented as m(t). The position of the vehicle determined by the application of the amount of operation is p(t), and similarly, the position of the vehicle determined by the application of the amount of operation M0(T0) in the exemplary operation is represented as P0(T0). The start time T0 of the exemplary operation and the start time t0 of the driver's actual operation are synchronized, and thereafter, for example, if the amount of operation in the exemplary operation at T1 is M0(T1), the amount of operation in the driver's actual operation can be represented as m(t0+T1-T0). These M0(T1) and m(t0+T1-T0) are calculated and used for comparison. After this, the process moves to step S211.

[0063] Next, in step S211, the transition operation calculation unit 105 compares the difference between the direction of movement of the vehicle in the model operation and the direction of movement of the own vehicle.

[0064] Specifically, the direction of movement can be calculated, for example, from the change in position over time. The direction of movement D0(T1) of the vehicle in the exemplary operation can be expressed as D0(T1)=(P0(T1)-P0(T0)) / (T1-T0). Similarly, the direction of movement d(t0+T1-T0) of the own vehicle can be expressed as d(t0+T1-T0)=p((t0+T1-T0)-p(t0)) / (T1-T0). These are compared, and if the difference is greater than a predetermined value Cd, it is determined that the difference in direction is large, and the process moves to step S213. Otherwise, the process moves to step S212.

[0065] Next, in step S212, the transition operation calculation unit 105 compares the difference between the vehicle's position P0(T1) in the model operation and the vehicle's own position p(t0+T1-T0). If this difference is greater than a predetermined value Cp, it is determined that the position difference is large, and the process moves to step S213. Otherwise, the process moves to step S215.

[0066] Next, in step S213, the transition operation calculation unit 105, based on the previous determinations, considers that there is a large difference between the behavior of the vehicle in the exemplary operation, such as its position and direction of movement, and the behavior of the own vehicle, such as its position and direction of movement, and calculates the amount of operation required to transition the behavior of the own vehicle to the behavior of the vehicle in the exemplary operation. This step S213 corresponds to the transition operation calculation step.

[0067] As an example, Figure 6 shows a situation where there is a large difference between the direction of the vehicle in the exemplary operation and the direction of the driver's own vehicle. Figure 6 is a diagram illustrating the change in the direction of vehicle movement in the driver operation and exemplary operation of the driver assistance device in Example 1.

[0068] As a specific scenario, assume that when turning left onto a road, vehicle 501 moves along its own trajectory 502.

[0069] In this case, the vehicle 503 in the model operation, extracted in step S209 based on the information acquired by the environmental information acquisition unit 101, shows movement along the trajectory 504 in the model operation.

[0070] Vehicle 501, while entering the road to turn left, made a wide turn to the right to avoid entering the oncoming lane. However, as a result, it entered the oncoming lane before the left turn, reducing traffic safety.

[0071] In contrast, in the demonstration, vehicle 503 moves slowly to the left, avoiding entering the oncoming lane both before and after the left turn.

[0072] Here, the difference between the direction d(t0+T2-T0) of the vehicle 501 at time T2 and the direction D0(T2) of the vehicle 503 in the exemplary operation becomes large and exceeds Cd, causing the process to branch in step S211 and proceed to step S213.

[0073] In step S213, the operation amount to transition from the point T2 when the difference in direction between the local vehicle 501 and the vehicle 503 in the exemplary operation becomes large, to the point when the difference in direction becomes less than or equal to Cd, i.e., in this example, time T4, is calculated.

[0074] Using the direction d(t0+T2-T0) and position p(t0+T2-T0) of the vehicle 501 at time T2, and the direction D0(T4) and position P0(T4) of the vehicle 503 during the exemplary operation at time T4, a clothoid curve is created, for example, connecting the two. The maneuver amounts when moving along this curve are represented by M0'(T2), M0'(T3), and M0'(T4). This is shown in Figure 7. Figure 7 is a diagram illustrating the flow of the trajectory from the vehicle's trajectory through the transition trajectory to the exemplary trajectory and the maneuver amounts in the driving assistance device of Embodiment 1.

[0075] As shown in Figure 7, the position and direction of the vehicle 501 at time points T0, T1, and T2 remain the same, and it moves along the vehicle's trajectory 502. However, the manipulated amount at time point T2 becomes M0'(T2), and the position and direction of the vehicle 501 change thereafter, moving along the transition trajectory 601. After further movement based on the manipulated amounts M0'(T3) and M0'(T4), the vehicle 501 is in the same position and direction as the vehicle 503 in the exemplary operation, and thereafter moves along the trajectory 504 in the exemplary operation. After this, the process moves to step S214.

[0076] Next, in step S214, the transition operation calculation unit 105 displays the resulting transition operation trajectory on the output unit 107 (display 906). This step S214 corresponds to the output step.

[0077] Figure 8 shows an example of the results output to the output unit 107. Figure 8 is a diagram illustrating the flow of the trajectory from the vehicle's trajectory to the model trajectory via the transition trajectory, the manipulated amount, and the display of the original vehicle's trajectory in the driving assistance device of Embodiment 1.

[0078] Specifically, based on the results calculated in the explanations of Figures 6 and 7, the road shape is drawn using environmental information acquired from the environmental information acquisition unit 101, and the vehicle's trajectory 502, the transition trajectory 601, and the trajectory 504 of the exemplary operation from time T4 onwards are drawn and output on top of it. In addition, the operation amounts m(t0) to M0(T5) when moving from the vehicle's trajectory 502 through the transition trajectory 601 to the trajectory 504 of the exemplary operation from time T4 onwards are output. This outputs the operation content and trajectory from the driver's own operation to the exemplary operation. At this time, the transition trajectory 601 can be highlighted.

[0079] Figure 9 shows another example of the results output to the output unit 107. Figure 9 is a diagram illustrating the flow of the trajectory from the vehicle's trajectory to the model trajectory via the transition trajectory in the driving assistance device of Embodiment 1, and the display of the vehicle's trajectory from the driver's perspective.

[0080] As shown in Figure 9, a video is created from the driver's perspective, and the shape of the road is drawn using environmental information acquired from the environmental information acquisition unit 101. The vehicle's trajectory 502 and transition trajectory 601 are then output on top of it. The shapes of the vehicle's steering wheel 801, accelerator pedal 802, and brake pedal 803 are also displayed at the bottom of the screen. Although the accelerator pedal 802 and brake pedal 803 are located at the driver's feet in a real vehicle, they are placed near the steering wheel 801 for easier viewing on the output unit 107.

[0081] The steering wheel 801, accelerator pedal 802, and brake pedal 803 of the vehicle each have symbols indicating the amount of operation in the exemplary operation and the amount of operation performed by the driver.

[0082] In Figure 9, the pentagonal symbol indicates the amount of operation in the exemplary operation (exemplary operation information), and the triangular symbol indicates the amount of operation performed by the driver themselves. For example, in the case of the vehicle's steering wheel 801, the angle of how much to the left or right from the center position is shown as an angle from the top of the steering wheel with the center of the steering wheel as the center. In the case of the vehicle's accelerator pedal 802 and brake pedal 803, if there is a symbol at the bottom of the displayed rectangle, it means no pedal depression, and if there is a symbol at the top, it means maximum pedal depression.

[0083] This facilitates the driver's understanding of how much they need to change their own actions to transition to the exemplary operation.

[0084] In this example, the display of these manipulated variables starts at time T0 and changes as time progresses to time T5. These displays could also be shown on an LCD screen or a VR (Virtual Reality) head-mounted display. After this, the process moves to step S215.

[0085] Next, in step S215, the operation information identification unit 102 determines whether it has processed all the information acquired by the driver while driving the vehicle. If processing is complete, the entire process ends; otherwise, the process moves to step S204.

[0086] Next, the effects of this embodiment will be described.

[0087] The driver assistance device 1 of Embodiment 1 of the present invention described above comprises: an environmental information acquisition unit 101 that acquires vehicle information and information about the surrounding environment of the vehicle; an operation information identification unit 102 that identifies operation information, which is information about actual operations performed by the driver, based on the surrounding environment information; an exemplary operation calculation unit 104 that calculates exemplary operations that the driver should perform based on the surrounding environment information; a transition operation calculation unit 105 that uses the operation information and exemplary operations to calculate exemplary operation information that transitions from actual operations to exemplary operations; a virtual space construction unit 106 that virtually constructs a space in which driving was performed based on the surrounding environment information; and an output unit 107 that outputs actual operation and exemplary operation information together with the space.

[0088] This allows drivers to see the amount of input and the trajectory (i.e., timing) required to transition from their actual driving actions to exemplary actions. As a result, they can identify actions that need improvement and actions after improvement, thereby promoting the improvement of drivers' driving skills.

[0089] The driver assistance device 1 of this embodiment can also be applied to existing driver assistance devices in the form of a software update.

[0090] Furthermore, the system includes a model operation database 103 for storing model operation quantities during operation. The transition operation calculation unit 105 calculates model operation information using the model operation quantity information stored in the model operation database 103. This allows for rapid calculation of model operation quantities, thus enabling faster processing.

[0091] Furthermore, the output unit 107 emphasizes the transition trajectory 601 that shows the transition from actual operation to exemplary operation, making it easier to understand what the exemplary operation was, and thus further promoting the improvement of the driver's driving skills.

[0092] <Example 2> The driver assistance device, driver assistance system, and driver assistance method of Embodiment 2 of the present invention will be explained with reference to Figures 10 and 11.

[0093] An example of the system configuration in this embodiment is shown in Figure 10. Figure 10 is a configuration diagram of the driver assistance device of Embodiment 2. The driver assistance device 1A of this embodiment shown in Figure 10 includes a trajectory calculation unit 108A that calculates the trajectory of a model operation based on surrounding environment information, instead of the model operation database 103 in the driver assistance device 1 shown in Figure 1.

[0094] The trajectory calculation unit 108A calculates the trajectory of movement along the road based on the information input from the environmental information acquisition unit 101 and outputs it to the model operation calculation unit 104.

[0095] Furthermore, the transition operation calculation unit 105A calculates exemplary operation information using the trajectory information calculated by the trajectory calculation unit 108A.

[0096] The processing flow in this embodiment will be explained.

[0097] The basic flow is similar to that shown in Figure 3, but only the processing in step S209 is different. The details of the processing in step S209 in this embodiment are as follows.

[0098] The trajectory calculation unit 108A calculates the trajectory and operation amount corresponding to the model operation based on the type of model operation determined in step S209 or earlier. An example when the type of model operation is determined to be a left turn is shown in Figure 11. Figure 11 is a diagram illustrating the flow of the trajectory and operation amount in the driving assistance device of Embodiment 2, from the vehicle's trajectory to the model trajectory via the transition trajectory.

[0099] In Figure 11, we assume that vehicle 1101 is moving according to the model operation calculated. Here, the trajectory of movement is divided into three parts.

[0100] The trajectory 1102 before the left turn is calculated from the road layout and width, which are environmental information input from the environmental information acquisition unit 101, and the vehicle width, which is vehicle information, as a trajectory that allows the vehicle to travel at a constant distance from the left edge of the road.

[0101] Similarly, the trajectory 1103 after the left turn is calculated as the trajectory that allows the vehicle to travel at a constant distance from the left edge of the road after the left turn.

[0102] The trajectory 1104 during the left turn is defined as the trajectory connecting the end of the trajectory 1102 before the left turn and the beginning of the trajectory 1103 after the left turn, and is calculated using, for example, a clothoid curve.

[0103] These are combined to calculate the trajectory, and the manipulated variables at each point in time are calculated as Mc0(T0) and Mc0(T1), and these are used to perform the processing from step S210 onward.

[0104] The other configurations and operations are substantially the same as those of the driver assistance device, driver assistance system, and driver assistance method described in Embodiment 1 above, and details are omitted.

[0105] In the driver assistance device, driver assistance system, and driver assistance method of Embodiment 2 of the present invention, substantially the same effects as those of the driver assistance device, driver assistance system, and driver assistance method of Embodiment 1 described above can be obtained.

[0106] Furthermore, the system includes a trajectory calculation unit 108A that calculates the trajectory of a model operation based on surrounding environment information. The transition operation calculation unit 105A calculates model operation information using the trajectory information calculated by the trajectory calculation unit 108A. This eliminates the need to pre-store model operations, such as those shown in Figure 5, in the model operation database 103, thus enabling the calculation of model operations quickly.

[0107] <Example 3> A driver assistance device, driver assistance system, and driver assistance method according to Embodiment 3 of the present invention will be explained with reference to Figure 12.

[0108] The driving support device of this embodiment displays the driving pattern resulting from the calculated transition operations from a different viewpoint, as shown in the driving support device 1 of Embodiment 1 or the driving support device 1A of Embodiment 2. The output unit 107 outputs the state of the vehicle at the time of transitioning to the model trajectory realized by the model operation, from a viewpoint that is inside the surrounding environment information and outside the vehicle.

[0109] Figure 12 shows the operation details and trajectory in this embodiment. Figure 12 is a diagram illustrating an example of displaying the operation process resulting from the transition operation in the driver assistance device of Embodiment 3 from a different perspective.

[0110] Figure 12 shows the vehicle's trajectory 502, the transition trajectory 601, and the trajectory 504 of the exemplary operation from time T4 onward, as described in the explanation of Figure 7. Within the entire trajectory shown in Figure 12, time T2, when the driver changes the amount of control made by the driver, is determined to be important for improving the driver's driving skills. At this time, the environmental information, vehicle positioning, and the entire trajectory are visualized and displayed from an external viewpoint so that they can be objectively understood. This is shown as a display content 1201 from a different viewpoint in the lower right of Figure 12.

[0111] The viewpoint for visualization is determined so that environmental information, the vehicle, and the trajectory are displayed, centered on the vehicle's position at time T2. By setting the viewpoint perpendicular to the vehicle's direction of movement, it becomes easier to understand the vehicle's movement. Alternatively, one could consider using the point in time where the change in the amount of control is greatest within the entire trajectory as the viewpoint.

[0112] The other configurations and operations are substantially the same as those of the driver assistance device, driver assistance system, and driver assistance method described in Embodiment 1 above, and details are omitted.

[0113] In the driver assistance device, driver assistance system, and driver assistance method of Embodiment 3 of the present invention, substantially the same effects as those of the driver assistance device, driver assistance system, and driver assistance method of Embodiment 1 described above can be obtained.

[0114] Furthermore, the output unit 107 outputs the vehicle's state at the point of transition to the model trajectory achieved by the model operation, from the perspective of both the surrounding environment information and the outside of the vehicle. This allows the driver to objectively understand how they transition from their own operations to the model operation, thereby further promoting the improvement of the driver's driving skills.

[0115] <Example 4> A driver assistance device, driver assistance system, and driver assistance method according to Embodiment 4 of the present invention will be described with reference to Figures 13 and 14.

[0116] The driving support device of this embodiment calculates the trajectory and operation amount to respond to new traffic conditions by creating a new scene in which virtual objects are placed relative to environmental information, in addition to the driving support system of any of Embodiments 1 to 3. The virtual space construction unit 106 further creates a new scene in which virtual objects are placed relative to the surrounding environment, and the model operation calculation unit 104 calculates the model trajectory and model operation to be realized by the model operation based on the new scene created by the virtual space construction unit 106.

[0117] An example of environmental information in this embodiment is shown in Figure 13. Figure 13 illustrates an example of calculating trajectories and operation amounts to respond to new traffic conditions by adding virtual objects to the driver assistance system of Embodiment 4.

[0118] As shown in Figure 13, the situation is shown in which the vehicle 501 is making a left turn. In actual driving by the driver, there are no other vehicles on the road after the left turn, but here a virtual vehicle 1301 is placed on the road after the left turn as an oncoming vehicle. This virtual vehicle 1301 and the environmental information input from the environmental information acquisition unit 101 are combined and then the exemplary operation extraction process is performed. At this time, the exemplary operation database 103 is assumed to hold exemplary operations as shown in Figure 14. Figure 14 is a diagram showing the exemplary operation information held by the exemplary operation database in the example of adding a virtual object in the driving assistance device of Embodiment 4.

[0119] In this example, since it is a left turn with oncoming traffic, M7(T0) and subsequent steps are extracted. Based on this, processing continues, and the updated model trajectory 1302, updated transition trajectory 1303, and manipulated variables m(t0), m(t0+T1-T0), M7'(T2), M7'(T3), M7'(T4), and M7(T5) are calculated to avoid contact with the oncoming vehicle.

[0120] The other configurations and operations are substantially the same as those of the driver assistance device, driver assistance system, and driver assistance method described in Embodiment 1 above, and details are omitted.

[0121] In the driver assistance device, driver assistance system, and driver assistance method of Embodiment 4 of the present invention, substantially the same effects as those of the driver assistance device, driver assistance system, and driver assistance method of Embodiment 1 described above can be obtained.

[0122] Furthermore, the virtual space construction unit 106 creates a new scene by placing virtual objects in the surrounding environment, and the exemplary operation calculation unit 104 calculates an exemplary trajectory and exemplary operation realized by the exemplary operation based on the scene. This makes it possible to calculate the amount of operation that takes into account not only the driver's own operations but also other traffic conditions, and by showing the driver the amount of operation in various traffic conditions, it is possible to further promote the improvement of the driver's driving skills.

[0123] <Example 5> An example of the driver assistance device, driver assistance system, and driver assistance method of the present invention will be described with reference to Figure 15.

[0124] The driver assistance device of this embodiment is a driver assistance system of any of Embodiments 1 to 4 that displays operation content and trajectories superimposed on the viewpoint in the actual environment using AR (Augmented Reality). The output unit 107 displays the vehicle, the actual trajectory realized by actual operation, the model trajectory realized by model operation, and the transition trajectory obtained from model operation information superimposed on the real landscape constructed from the surrounding environment information acquired by the environment information acquisition unit.

[0125] Figure 15 shows an example of the content displayed on the AR head-mounted display in this embodiment. Figure 15 is a diagram illustrating an example of displaying operation content and trajectory superimposed on the viewpoint in the actual environment in the driver assistance device of Embodiment 5.

[0126] As shown in Figure 15, AR makes it possible to superimpose virtual objects onto real-world scenery. The AR head-mounted display detects the user's position and orientation, and the environmental information input from the environmental information acquisition unit 101 is matched to perform the synthesis of the two. In this example, a virtual vehicle 1502, a virtual vehicle trajectory 1503, and a virtual composite trajectory 1504 (a combination of the vehicle's trajectory, transition trajectory, and model trajectory) are superimposed onto the real-world scenery 1501, which includes roads and ground.

[0127] The other configurations and operations are substantially the same as those of the driver assistance device, driver assistance system, and driver assistance method described in Embodiment 1 above, and details are omitted.

[0128] In the driver assistance device, driver assistance system, and driver assistance method of Embodiment 5 of the present invention, substantially the same effects as those of the driver assistance device, driver assistance system, and driver assistance method of Embodiment 1 described above can be obtained.

[0129] Furthermore, the output unit 107 displays the vehicle, the actual trajectory realized by actual operation, the model trajectory realized by model operation, and the transition trajectory obtained from model operation information overlaid on the real landscape constructed from the surrounding environmental information acquired by the environmental information acquisition unit. By displaying the trajectory in the real space in which the driver is actually driving and allowing the driver to check it from various perspectives, it is possible to further promote the improvement of the driver's driving skills.

[0130] <Other> It should be noted that the present invention is not limited to the embodiments described above, and includes various modifications. The embodiments described above are explained in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those having all the configurations described.

[0131] Furthermore, it is possible to replace parts of the configuration of one embodiment with parts of the configuration of another embodiment, and it is also possible to add parts of the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with parts of other configurations. [Explanation of symbols]

[0132] 1,1A...Driving support system 101…Environmental information acquisition department 102... Operation Information Identification Unit 103... Exemplary Operation Database 104...Model operation calculation section 105,105A…Transition operation calculation unit 106…Virtual Space Construction Department 107...Output section 108A…Trajectory calculation unit 501... My car 502... My vehicle's trajectory 503... Vehicles in exemplary operation 504... Trajectory in exemplary operation 601…Transition trajectory 801... Vehicle steering wheel 802... Vehicle accelerator pedal 803... Vehicle brake pedal 901...Data bus 902...Sensor 903...Storage device 904…CPU 905...RAM 906…display 1101... Vehicle in exemplary operation 1102...Trajectory before turning left 1103...Trajectory after turning left 1104...Trajectory while turning left 1201... Display content from a different perspective 1301... Virtual vehicle 1302…Updated model trajectory 1303…Update transition trajectory 1501...Actual scenery 1502...Virtual vehicle 1503... Virtual trajectory of the vehicle 1504...Virtual composite trajectory

Claims

1. An environmental information acquisition unit that acquires mobile information about a mobile object and information about the surrounding environment of the mobile object, An operation information identification unit identifies operation information, which is information relating to the actual operation performed by the driver, based on the surrounding environment information. Based on the surrounding environment information, the system includes a model operation calculation unit that calculates a model operation that the driver should perform, A transition operation calculation unit calculates model operation information that transitions from the actual operation to the model operation using the aforementioned operation information and the model operation, A virtual space construction unit that virtually constructs the space in which the operation was performed based on the aforementioned surrounding environment information, The system includes an output unit that outputs the aforementioned operation information and the aforementioned exemplary operation information to the space. Driving assistance system.

2. In the driving support device according to claim 1, It also includes a model operation database that stores exemplary operation amounts in driving, The transition operation calculation unit calculates the model operation information using the model operation quantity information stored in the model operation database. Driving assistance system.

3. In the driving support device according to claim 1, The system further includes a trajectory calculation unit that calculates the trajectory of the model operation based on the surrounding environment information, The transition operation calculation unit calculates the model operation information using the trajectory information calculated by the trajectory calculation unit. Driving assistance system.

4. In the driving support device according to claim 1, The output unit outputs the state of the moving object at the point in time when it transitions to the model trajectory realized by the model operation, from a perspective that is inside the surrounding environment information and outside the moving object. Driving assistance system.

5. In the driving support device according to claim 1, The virtual space construction unit further creates a new scene in which virtual objects are placed in the surrounding environment. The model operation calculation unit calculates the model trajectory and the model operation to be realized by the model operation based on the scene. Driving assistance system.

6. In the driving support device according to claim 3, The output unit overlays the moving object, the actual trajectory realized by the actual operation, the model trajectory realized by the model operation, and the transition trajectory obtained from the model operation information onto the real landscape constructed from the surrounding environment information acquired by the environment information acquisition unit. Driving assistance system.

7. In the driving support device according to claim 1, The output unit highlights the transition operation that transitions from the actual operation to the model operation. Driving assistance system.

8. The driving support device according to claim 1, A mobile body having a display, Driver assistance system.

9. The driving support device according to claim 1, Equipped with a display, A mobile object.

10. An environmental information acquisition step involves acquiring information about a mobile object and information about the surrounding environment of the mobile object. An operation information identification step identifies operation information, which is information about the actual operation performed by the driver, based on the surrounding environment information. A model operation calculation step that calculates a model operation that the driver should perform based on the surrounding environment information, A transition operation calculation step that calculates model operation information that transitions from the actual operation to the model operation using the aforementioned operation information and the model operation, A virtual space construction step in which the space in which the operation was performed is virtually constructed from the surrounding environment information, The system includes an output step that outputs the aforementioned operation information and the aforementioned exemplary operation information together in the space. Driving assistance methods for automobiles.

Images